GitHub repository: scikit-learn/scikit-learn

Issues:
issue_28561: state: open
title: DOC Add a docstring for sklearn.utils.random._random_choice_csc
body: Fixes parts of #27982

Added a docstring example for the following random function:

- [ ] sklearn.utils.random._random_choice_csc

user: divakshu04
issue_28560: state: open
title: CI Use shared tool to upload nightly build wheels 
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Closes #24312
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
Use [scientific-python shared tool](http://example.com) to upload nightly build wheels 
#### Any other comments?
To be tested

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
https://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: lamdang2k
issue_28559: state: open
title: Gaussian mixture lower bounds
body: #### Reference Issues/PRs
Add new feature #28492

#### What does this implement/fix? Explain your changes.
The new feature adds the ability to store and access the list of lower bounds for each iteration in the BaseMixture class, providing insights into the convergence behavior of mixture models like GaussianMixture.

#### Any other comments?
Here's a code sample for using it on Iris dataset:
```
from sklearn import datasets
from sklearn.mixture import GaussianMixture
import matplotlib.pyplot as plt
import pandas as pd

iris = datasets.load_iris()
X = iris.data

d = pd.DataFrame(X)

gmm = GaussianMixture(n_components=3, n_init=10)
gmm.fit(d)

lower_bounds = gmm.lower_bounds_
n_iter = gmm.n_iter_

plt.figure(figsize=(10, 6))
plt.plot(range(1, n_iter + 1), lower_bounds, marker='o', linestyle='-')
plt.title('GMM Lower Bounds Across Iterations')
plt.xlabel('Iteration')
plt.ylabel('Lower Bound')
plt.grid(True)
plt.show()
``` 

<img width="980" alt="image" src="https://github.com/scikit-learn/scikit-learn/assets/127900888/4fb05fc7-9779-4499-b432-1afc64828845">

user: myenugula
issue_28558: state: open
title: Inaccurate Attribute Listing with dir(obj) for Classes Using available_if Conditional Method Decorator
body: ### Describe the bug

When utilizing the `available_if` decorator from SciKit Learn to conditionally expose methods based on specific object state or conditions, we observe that the `dir(obj)` function may return inaccurate results. Specifically, `dir(obj)` continues to list methods that should be conditionally hidden based on the `available_if` decorator's logic. This discrepancy arises because the `__dir__` method on the affected classes does not dynamically account for this conditional availability. As a result, users and consuming code may be misled about the actual methods available for use on instances of the class at runtime, potentially leading to unexpected `AttributeErrors` when accessing supposedly available methods.

### Steps/Code to Reproduce

I will test with the SVC, but it can apply to other classes.

```python
from sklearn.svm import SVC
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

model = SVC(probability=False)
model.fit(X[:100], y[:100])

# Check if 'predict_proba' is listed by dir()
print("'predict_proba' in dir(model):", "predict_proba" in dir(model))

# Attempt to call 'predict_proba'
try:
    prob_predictions = model.predict_proba(X[:2])
    print("Predict_proba called successfully.")
except AttributeError as e:
    print("Attempting to call 'predict_proba' raised an AttributeError:", e)

```

### Expected Results

It should print out the following:
```
'predict_proba' in dir(model): False
Attempting to call 'predict_proba' raised an AttributeError: predict_proba is not available when probability=False
```

Methods decorated with `available_if` and whose conditions raise an `AttributeError` should not appear in the list returned by `dir(obj)`.

### Actual Results

It should print out the following:
```
'predict_proba' in dir(model): True
Attempting to call 'predict_proba' raised an AttributeError: predict_proba is not available when probability=False
```

Methods decorated with `available_if` and whose conditions raise an `AttributeError` appears in the list returned by `dir(obj)`.

Traceback when you show the error itself (comes from the check in `available_if`.
```
Traceback (most recent call last):
  File "/Users/kmcgrady/Projects/test-scripts/xg_example.py", line 48, in <module>
    prob_predictions = model.predict_proba(X[:2])
  File "/Users/kmcgrady/.local/share/virtualenvs/test-scripts-fLg6VU9n/lib/python3.10/site-packages/sklearn/utils/_available_if.py", line 31, in __get__
    if not self.check(obj):
  File "/Users/kmcgrady/.local/share/virtualenvs/test-scripts-fLg6VU9n/lib/python3.10/site-packages/sklearn/svm/_base.py", line 827, in _check_proba
    raise AttributeError(
AttributeError: predict_proba is not available when probability=False
```

### Versions

```shell
System:
    python: 3.10.10 (main, May 13 2023, 16:09:51) [Clang 14.0.3 (clang-1403.0.22.14.1)]
executable: /Users/kmcgrady/.local/share/virtualenvs/test-scripts-fLg6VU9n/bin/python
   machine: macOS-14.3.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.3.2
          pip: 23.3.1
   setuptools: 68.2.2
        numpy: 1.26.4
        scipy: 1.11.4
       Cython: None
       pandas: 2.2.1
   matplotlib: 3.8.2
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 10
         prefix: libomp
       filepath: /Users/kmcgrady/.local/share/virtualenvs/test-scripts-fLg6VU9n/lib/python3.10/site-packages/sklearn/.dylibs/libomp.dylib
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 10
         prefix: libopenblas
       filepath: /Users/kmcgrady/.local/share/virtualenvs/test-scripts-fLg6VU9n/lib/python3.10/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: armv8

       user_api: blas
   internal_api: openblas
    num_threads: 10
         prefix: libopenblas
       filepath: /Users/kmcgrady/.local/share/virtualenvs/test-scripts-fLg6VU9n/lib/python3.10/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: armv8
```

user: kmcgrady
labels: Needs Triage: Issue requires triage
issue_28557: state: open
title: Fix `n_nonzero_coefs_` in `OrthogonalMatchingPursuit` always `None` when ignored
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
closes #28469

#### What does this implement/fix? Explain your changes.
Ensures that `n_nonzero_coefs_` in `OrthogonalMatchingPursuit` is always `None` when ignored. Clarified `n_nonzero_coefs` and `n_nonzero_coefs_` docstrings

#### Any other comments?

cc @StefanieSenger you may be interested in reviewing?
<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
https://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: lucyleeow
issue_28555: state: open
title: MAINT: convert `numpy.array_api` to array-api-strict
body: The `numpy.array_api` module has been converted into a standalone package (`array-api-strict`). This new package is stable and has had a 1.0 release. The `numpy.array_api` module was marked experimental, and will be removed for NumPy 2.0. Since `array-api-strict` works with both NumPy 1.2x and NumPy 2.0, using that for testing compliance to the syntax and semantics of the array API standard should always be preferred over still trying to use `numpy.array_api`.


#### Reference Issues/PRs

No separate scikit-learn issue for this; [NEP 56](https://numpy.org/neps/nep-0056-array-api-main-namespace.html) documents the decision to remove `numpy.array_api` before the NumPy 2.0 release

EDIT: https://github.com/numpy/numpy/pull/25911 is the relevant NumPy PR.


#### What does this implement/fix? Explain your changes.

This is mostly a 1:1 replacement of `numpy.array_api` with `array_api_strict`. The improvements in the latter package did turn up a couple of places where the code wasn't compliant with the standard though, due to:
- using a `device='cpu'` string which was NumPy/PyTorch-specific,
- indexing non-numpy arrays with an integer `numpy.ndarray` (this should be using the same array types and use the `take` function instead of fancy indexing),
- use of `dtype.kind`, which isn't guaranteed to exist or return one letter numpy-style type codes,
- use of the builtin `int` as a dtype


#### Any other comments?

I have tested with:
- `numpy` 1.26.4
- `array-api-strict` 1.0
- `pytorch` 2.1.2 (CPU and CUDA 12.3)
- `cupy` 13.0.0 (CUDA 12.3)

The only package with failures was CuPy. I'm seeing 35 failures on `main`, and 24 with the changes in this PR. Remaining CuPy failures are from
1. `test_array_api.py::test_nan_reductions` (20x) with these errors:
```
TypeError: Implicit conversion to a NumPy array is not allowed. Please use `.get()` to construct a NumPy array explicitly

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
```
2. `model_selection/tests/test_split.py::test_array_api_train_test_split` (4x) with these errors:
```
FAILED sklearn/model_selection/tests/test_split.py::test_array_api_train_test_split[True-None-cupy.array_api-None-None] - IndexError: Single-axes index [9 1 6 7 3 0 5] is a non-zero-dimensional integer array, but advanced integer indexing is not s...
FAILED sklearn/model_selection/tests/test_split.py::test_array_api_train_test_split[True-stratify1-cupy-None-None] - ValueError: kind can only be None or 'stable'
FAILED sklearn/model_selection/tests/test_split.py::test_array_api_train_test_split[True-stratify1-cupy.array_api-None-None] - ValueError: The least populated class in y has only 1 member, which is too few. The minimum number of groups for any class ca...
FAILED sklearn/model_selection/tests/test_split.py::test_array_api_train_test_split[False-None-cupy.array_api-None-None] - IndexError: Single-axes index [0 1 2 3 4 5 6] is a non-zero-dimensional integer array, but advanced integer indexing is not s...
```

Failures visible on `main` that are resolved with this PR are due to:
```
ValueError: Unsupported device 'cpu'
```
in the following tests:
- `utils/tests/test_array_api.py::test_weighted_sum`
- `tests/test_common.py::test_estimators[LinearDiscriminantAnalysis()-check_array_api_input`
- `metrics/tests/test_common.py::test_array_api_compliance`
user: rgommers
issue_28552: state: open
title: VIZ, ENH Adding True/False labels in arrows coming from root in `plot_tree` to match `export_tree`
body: #### Reference Issues/PRs
Fixes: https://github.com/scikit-learn/scikit-learn/issues/16153


#### What does this implement/fix? Explain your changes.
- Adds `True/False` labels on top of the arrows that go from root node to left/right child
- Updates unit-test


#### Any other comments?

The image from `plot_tree`:

![image](https://github.com/scikit-learn/scikit-learn/assets/3460267/3c3b522c-73f7-4d92-9f12-77a0cd643022)


user: adam2392
issue_28551: state: open
title: Implement `SplineTransformer.inverse_transform`
body: ### Describe the workflow you want to enable

I think it should be possible to implement a new method `inverse_transform` such that:

```python
import numpy as np
from sklearn.preprocessing import SplineTransformer

rng = np.random.default_rng(0)
X_train = rng.normal(size=(42, 5))
X_test = rng.normal(size=(43, 5))

st = SplineTransformer().fit(X_train)
np.testing.assert_allclose(X_test, st.inverse_transform(st.transform(X_test)))
```

### Describe your proposed solution


There might be several mathematical ways to define such a transform, in particular if when passing a `X_fake_transformed` that contain real numbers that do not actually result from a spline expansion. For instance when:

- `(X_fake_transformed < 0).any()`
- `(X_fake_transformed > 1).any()`
- `X_fake_transformed.sum(axis=1) != np.ones(n_samples)`.

or when all values of a given row are non-zeros at once...

One possible way would be to decode based on `X_fake_transformed.argmax(axis=1)` and then using the relative strength of neighboring spline activations to resolve ambiguities.

### Describe alternatives you've considered, if relevant

The main alternative is to not implement this. The main question is probably why try to implement this in the first place?

Possible use cases:

- fit a GMM model on spline transformed data (to get a more axis-aligned inductive prior), generate samples in the GMM latent space and then recode those samples back into the original space.

- fit PCA with a small rank on spline encoded data and then reconstruct back the projected data,

- fit k-means in spline space and recode the learned centroids back in the original feature space for inspection.

- idem for NMF or dictionary learning components.


### Additional context

If #28043 gets merged, missing values support should also be included when using the 'indicator' strategy.
user: ogrisel
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_28550: state: open
title: DOC Iter on outlier detection estimators example
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Follow-up of #25878

#### What does this implement/fix? Explain your changes.

Addresses the suggestions made by @ogrisel in https://github.com/scikit-learn/scikit-learn/pull/25878#pullrequestreview-1361057671.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
https://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: ArturoAmorQ
labels: Quick Review: For PRs that are quick to review
issue_28549: state: open
title: Make pipeline cache ignore parameter `verbose` of transformers
body: ### Describe the workflow you want to enable

**Introduction**

sklearn's pipeline caches the output of transformers in the pipeline. The caching is based on a hash of the arguments of function `_fit_transform_one`. Unfortunately, the hash changes when **any** of the transformer's parameters change, including those that don't affect the output, for example the `verbose` parameter (there could be other ones, perhaps `copy`?).

It would be a nice feature if there would be a way to indicate to the pipeline (or if the pipeline can detect it automatically) which parameters within the transformers to ignore for caching.

**Use case**

While developing, I tend to always set a high verbosity to understand what's happening under the hood. Once I am content with the results, I turn the verbosity off. At this point, the results are already calculated and cached, but need to be recalculated because the change of parameter.

**Examples of affected transformers**

* ColumnTransformer
* RFE
* RFECV
* SparsePCA
* IterativeImputer


### Describe your proposed solution

The pipeline's caching is performed here:

https://github.com/scikit-learn/scikit-learn/blob/38b39a403179dd67b325dc2fe3da849feda7f557/sklearn/pipeline.py#L392

It uses joblib's [`Memory.cache`](https://joblib.readthedocs.io/en/latest/generated/joblib.Memory.html#joblib.Memory.cache), which accepts an `ignore` parameter to ignore arguments in the hashing, but you can't ignore parameters within one of the arguments (the first argument to `_fit_transform_one` is the transformer and we would like to ignore `verbose` within arg 1).

I couldn't find any trivial solution without monkey patching joblib's code or programmatically changing the `verbose` parameter in the transformers self (which would lead to unexpected results for the user, for example the lack of output messages). Happy to open a PR if anyone can think of a solution.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Similar to #23788 but presenting a new use case, which as far as I know, can't be resolved now.
user: tvdboom
labels: Needs Triage: Issue requires triage
issue_28548: state: open
title: Multiclass support in precision_recall_curve
body: ### Describe the workflow you want to enable

i would like to add multiclass support to precision_recall_curve.

### Describe your proposed solution

- Add check in the beginning to check if multiclass or binary
- Add weighting argument for `micro`, `macro`, `weighted`
- Implement _multiclass_clf_curve

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I can implement the functionality, but I would like to hear any comments before starting
user: isacarnekvist
labels: Needs Triage: Issue requires triage
issue_28547: state: open
title: Localization of scikit-learn website content.
body: Hi,

I work for Quansight Labs and am helping set things up for translation of main project website content for core projects in the Scientific Python ecosystem. This work is supported by the [Scientific Python Community & Communications Infrastructure grant](https://scientific-python.org/doc/scientific-python-community-and-communications-infrastructure-2022.pdf) from CZI. I created a Github discussion for this here: https://github.com/scikit-learn/scikit-learn/discussions/28105, but @glemaitre pointed out that an issue would be more appropriate.

A deliverable for this grant is to have the brochure websites of at least 8 of the 10 Scientific Python core projects translated into at least 3 commonly used languages. You may have seen the language drop-down selector at https://numpy.org/.  The goal is for a cross functional team of Quansight staff and volunteers to handle the bulk of the work, taking the burden off of project maintainers.

The localization management platform [Crowdin](https://crowdin.com/) is offering a free supported enterprise organization for Scientific Python translations. This is the platform we used for numpy.org. At this moment you would not need to decide what if anything you would do with translated content. *However, to get things started, I'd like to ask permission to fill out the [Crowdin Open Source Project Setup Request ](https://crowdin.com/page/open-source-project-setup-request) form on your behalf, to allow me to add a project to the Scientific Python Crowdin organization for Scikit-Learn.* 

For those interested. Here are the specifics of what I'd do with this.

- Create a GitHub repository which mirrors the content from the Scikit-Learn brochure website.
- Setup a cron github action which polls for updates to the Scikit-Learn website content and helps keep the mirror up to date.
- Sync this repository to the Scikit-Learn Crowdin project. Translate can then translate the content, and Crowdin will automatically push commits to a PR against the repository mirroring the website content.

The end product will be a repository with parallel versions of the website content in a variety of languages which is kept up to date. If you want to host the translations on your webpage like NumPy does, myself and/or a colleague from Quansight would be supported by the grant to help make that happen.

Please let me know if you have any questions.
user: steppi
labels: Needs Triage: Issue requires triage
issue_28546: state: open
title: MAINT cleanup utils.__init__: move indexing tools into dedicated submodule
body: Extracted from https://github.com/scikit-learn/scikit-learn/pull/26686 to ease the reviews. The end goal is to clean the `utils.__init__` module as explained in the linked PR.

I didn't find a relevant already existing module to move the functions related to indexing and assigning so I created a dedicated `utils._indexing` submodule.

`resample` and `shuffle` are public so I kept them importable from utils. These 2, I'm not sure it's the appropriate submodule for them but it's not bad either. Happy to take better suggestions.

`_safe_indexing` is also public although it has a leading underscore, but I kept as is.

The other ones are private so I made them only importable from `utils._indexing`.


user: jeremiedbb
issue_28541: state: open
title: GCC 14: Remove `inline` for `rdist_csr` and `rdist`
body: 
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes https://github.com/scikit-learn/scikit-learn/issues/28530

Relates to cython/cython#2747

Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=2261602

#### What does this implement/fix? Explain your changes.

This problem was mentioned in a bug open to keep track of the effort to remove all compilation warnings #24875 (search "Incompatible pointers types casts")

With gcc < 14 the code compiles with incompatible-pointer warning. But with gcc 14 that warning is an error.

A possible workaround is to remove the inline in the definition of the `rdist_csr` and `rdist` members.

With that change, cython generates correct code (I don't know why). But removing the inline can impact the performance.


#### Any other comments?

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
https://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: hswong3i
issue_28536: state: open
title: ValidationCurveDisplay can't handle categorical/string parameters
body: ### Describe the bug

Hi,

I performed some optimization on a few models implemented via the sklearn API. For fine tuning, I want to visualize the effect of certain hyperparameters using the `ValidationCurveDisplay` implementation. For numerical parameters, everything works fine. Unfortunately, as soon as categorical parameters (passed as strings) are used, an error is raised.


### Steps/Code to Reproduce


```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import ValidationCurveDisplay, validation_curve
from sklearn.linear_model import LogisticRegression

X, y = make_classification(n_samples=1_000, random_state=0)
logistic_regression = LogisticRegression()

# put categorical values in param space
param_name, param_range = "penalty", ['elasticnet', 'l1', 'l2']
train_scores, test_scores = validation_curve(
     logistic_regression, X, y, param_name=param_name, param_range=param_range
 )
display = ValidationCurveDisplay(
     param_name=param_name, param_range=param_range,
     train_scores=train_scores, test_scores=test_scores, score_name="Score"
 )
display.plot()
plt.show()
```

### Expected Results

The expected result is a validation curve display separating the values by their category and no errors.

### Actual Results

```python
---------------------------------------------------------------------------
UFuncTypeError                            Traceback (most recent call last)
Cell In [12], line 19
     12 train_scores, test_scores = validation_curve(
     13      logistic_regression, X, y, param_name=param_name, param_range=param_range
     14  )
     15 display = ValidationCurveDisplay(
     16      param_name=param_name, param_range=param_range,
     17      train_scores=train_scores, test_scores=test_scores, score_name="Score"
     18  )
---> 19 display.plot()
     20 plt.show()

File ~/work/miniconda/envs/GC_overhaul_nb/lib/python3.10/site-packages/sklearn/model_selection/_plot.py:691, in ValidationCurveDisplay.plot(self, ax, negate_score, score_name, score_type, std_display_style, line_kw, fill_between_kw, errorbar_kw)
    630 def plot(
    631     self,
    632     ax=None,
   (...)
    640     errorbar_kw=None,
    641 ):
    642     """Plot visualization.
    643 
    644     Parameters
   (...)
    689         Object that stores computed values.
    690     """
--> 691     self._plot_curve(
    692         self.param_range,
    693         ax=ax,
    694         negate_score=negate_score,
    695         score_name=score_name,
    696         score_type=score_type,
    697         log_scale="deprecated",
    698         std_display_style=std_display_style,
    699         line_kw=line_kw,
    700         fill_between_kw=fill_between_kw,
    701         errorbar_kw=errorbar_kw,
    702     )
    703     self.ax_.set_xlabel(f"{self.param_name}")
    704     return self

File ~/work/miniconda/envs/GC_overhaul_nb/lib/python3.10/site-packages/sklearn/model_selection/_plot.py:126, in _BaseCurveDisplay._plot_curve(self, x_data, ax, negate_score, score_name, score_type, log_scale, std_display_style, line_kw, fill_between_kw, errorbar_kw)
    121     xscale = "log" if log_scale else "linear"
    122 else:
    123     # We found that a ratio, smaller or bigger than 5, between the largest and
    124     # smallest gap of the x values is a good indicator to choose between linear
    125     # and log scale.
--> 126     if _interval_max_min_ratio(x_data) > 5:
    127         xscale = "symlog" if x_data.min() <= 0 else "log"
    128     else:

File ~/work/miniconda/envs/GC_overhaul_nb/lib/python3.10/site-packages/sklearn/utils/_plotting.py:97, in _interval_max_min_ratio(data)
     90 def _interval_max_min_ratio(data):
     91     """Compute the ratio between the largest and smallest inter-point distances.
     92 
     93     A value larger than 5 typically indicates that the parameter range would
     94     better be displayed with a log scale while a linear scale would be more
     95     suitable otherwise.
     96     """
---> 97     diff = np.diff(np.sort(data))
     98     return diff.max() / diff.min()

File ~/work/miniconda/envs/GC_overhaul_nb/lib/python3.10/site-packages/numpy/lib/function_base.py:1452, in diff(a, n, axis, prepend, append)
   1450 op = not_equal if a.dtype == np.bool_ else subtract
   1451 for _ in range(n):
-> 1452     a = op(a[slice1], a[slice2])
   1454 return a

UFuncTypeError: ufunc 'subtract' did not contain a loop with signature matching types (dtype('<U10'), dtype('<U10')) -> None
```

### Versions

```shell
System:
    python: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:35:26) [GCC 10.4.0]
executable: ~/work/miniconda/envs/GC_overhaul_nb/bin/python
   machine: Linux-4.18.0-348.2.1.el8_5.x86_64-x86_64-with-glibc2.28

Python dependencies:
      sklearn: 1.4.1.post1
          pip: 22.2.2
   setuptools: 65.4.1
        numpy: 1.25.2
        scipy: 1.9.2
       Cython: 3.0.0
       pandas: 1.5.0
   matplotlib: 3.8.0
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: ~/work/miniconda/envs/GC_overhaul_nb/lib/libopenblasp-r0.3.21.so
        version: 0.3.21
threading_layer: pthreads
   architecture: Haswell
    num_threads: 6

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: ~/work/miniconda/envs/GC_overhaul_nb/lib/libgomp.so.1.0.0
        version: None
    num_threads: 6
```

user: sroener
issue_28535: state: open
title: Add metrics.gini_index_score()
body: ### Describe the workflow you want to enable

The [Gini index](https://en.wikipedia.org/wiki/Gini_coefficient) metric (that is based on [Lorenz curve](https://en.wikipedia.org/wiki/Lorenz_curve)) is widely used in the insurance industry for evaluating the performance (ranking power) of various risk models.

### Describe your proposed solution

2 sklearn examples already includes code for calculating the gini index metric [here](https://github.com/scikit-learn/scikit-learn/blob/4e8253703013b38da503e2354d82fb7fa43dd4ec/examples/linear_model/plot_poisson_regression_non_normal_loss.py#L517) and [here](https://github.com/scikit-learn/scikit-learn/blob/4e8253703013b38da503e2354d82fb7fa43dd4ec/examples/linear_model/plot_tweedie_regression_insurance_claims.py#L676)

Related to #28534

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: yonil7
labels: Needs Triage: Issue requires triage
issue_28534: state: open
title: Two different versions for weighted lorenz curve calculation in the examples
body: ### Describe the issue linked to the documentation

There are 2 definitions of (weighted) `lorenz_curve()` functions [here](https://scikit-learn.org/stable/auto_examples/linear_model/plot_tweedie_regression_insurance_claims.html) and [here](https://scikit-learn.org/stable/auto_examples/linear_model/plot_poisson_regression_non_normal_loss.html)

The difference is in the X coordinates that these functions returns. Both return X coordinates between 0 and 1, but the first example returns **equally spaced** X coordinates:
```python
cumulated_samples = np.linspace(0, 1, len(cumulated_claim_amount))
```
and the second example return **un-equally spaced** X coordinates (spaced using the samples weights):
```python
cumulated_exposure = np.cumsum(ranked_exposure)
cumulated_exposure /= cumulated_exposure[-1]
```

### Suggest a potential alternative/fix

_No response_
user: yonil7
issue_28531: state: open
title: ⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️
body: **CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=64598&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Mar 01, 2024)
Unable to find junit file. Please see link for details.
user: scikit-learn-bot
labels: Needs Triage: Issue requires triage
issue_28530: state: open
title: BUILD gcc14 cannot compile scikit-learn
body: ### Describe the bug

When building 1.3.2+ with Fedora Rawhide at OBS, it is now failed with below error message (see https://build.opensuse.org/package/live_build_log/home:alvistack/scikit-learn-scikit-learn-1.4.1+post1/Fedora_Rawhide/x86_64):
```
running build_ext
building 'sklearn.metrics._dist_metrics' extension
gcc -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fcf-protection -fexceptions -fcf-protection -fexceptions -fcf-protection -fexceptions -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/usr/local/lib64/python3.12/site-packages/numpy/core/include -I/usr/include/python3.12 -c sklearn/metrics/_dist_metrics.c -o build/temp.linux-x86_64-cpython-312/sklearn/metrics/_dist_metrics.o -g0 -O2 -fopenmp
sklearn/metrics/_dist_metrics.c: In function ‘__pyx_pf_7sklearn_7metrics_13_dist_metrics_16DistanceMetric64_22_pairwise_sparse_dense’:
sklearn/metrics/_dist_metrics.c:29086:29: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
29086 |             __pyx_v_x2_data = ((&(*((__pyx_t_7sklearn_5utils_9_typedefs_float64_t const  *) ( /* dim=1 */ ((char *) (((__pyx_t_7sklearn_5utils_9_typedefs_float64_t const  *) ( /* dim=0 */ (__pyx_v_Y_data.data + __pyx_t_18 * __pyx_v_Y_data.strides[0]) )) + __pyx_t_19)) )))) + (__pyx_v_i2 * __pyx_v_n_features));
      |                             ^
sklearn/metrics/_dist_metrics.c: In function ‘__pyx_pf_7sklearn_7metrics_13_dist_metrics_16DistanceMetric64_24_pairwise_dense_sparse’:
sklearn/metrics/_dist_metrics.c:29871:27: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
29871 |           __pyx_v_x1_data = ((&(*((__pyx_t_7sklearn_5utils_9_typedefs_float64_t const  *) ( /* dim=1 */ ((char *) (((__pyx_t_7sklearn_5utils_9_typedefs_float64_t const  *) ( /* dim=0 */ (__pyx_v_X_data.data + __pyx_t_15 * __pyx_v_X_data.strides[0]) )) + __pyx_t_16)) )))) + (__pyx_v_i1 * __pyx_v_n_features));
      |                           ^
sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19EuclideanDistance64_dist_csr’:
sklearn/metrics/_dist_metrics.c:31532:90: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19EuclideanDistance64_rdist_csr’ from incompatible pointer type [-Wincompatible-pointer-types]
31532 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_19EuclideanDistance64_rdist_csr(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *)__pyx_v_self), __pyx_v_x1_data, __pyx_v_x1_indices, __pyx_v_x2_data, __pyx_v_x2_indices, __pyx_v_x1_start, __pyx_v_x1_end, __pyx_v_x2_start, __pyx_v_x2_end, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float64_t)-1.0))) __PYX_ERR(1, 1077, __pyx_L1_error)
      |                                                                                         ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                          |
      |                                                                                          struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *
sklearn/metrics/_dist_metrics.c:31165:213: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_EuclideanDistance64 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *’
31165 | csr(CYTHON_UNUSED struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_EuclideanDistance64 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x1_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x1_indices, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x2_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x2_indices, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_end, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_end, CYTHON_UNUSED __pyx_t_7sklearn_5utils_9_typedefs_intp_t const __pyx_v_size) {
      |                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_20SEuclideanDistance64_dist’:
sklearn/metrics/_dist_metrics.c:32037:87: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_20SEuclideanDistance64_rdist’ from incompatible pointer type [-Wincompatible-pointer-types]
32037 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_20SEuclideanDistance64_rdist(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *)__pyx_v_self), __pyx_v_x1, __pyx_v_x2, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float64_t)-1.0))) __PYX_ERR(1, 1126, __pyx_L1_error)
      |                                                                                      ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                       |
      |                                                                                       struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *
sklearn/metrics/_dist_metrics.c:31938:197: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_SEuclideanDistance64 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *’
31938 | Distance64_rdist(struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_SEuclideanDistance64 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x1, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x2, __pyx_t_7sklearn_5utils_9_typedefs_intp_t __pyx_v_size) {
      |                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_20SEuclideanDistance64_dist_csr’:
sklearn/metrics/_dist_metrics.c:32852:91: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_20SEuclideanDistance64_rdist_csr’ from incompatible pointer type [-Wincompatible-pointer-types]
32852 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_20SEuclideanDistance64_rdist_csr(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *)__pyx_v_self), __pyx_v_x1_data, __pyx_v_x1_indices, __pyx_v_x2_data, __pyx_v_x2_indices, __pyx_v_x1_start, __pyx_v_x1_end, __pyx_v_x2_start, __pyx_v_x2_end, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float64_t)-1.0))) __PYX_ERR(1, 1206, __pyx_L1_error)
      |                                                                                          ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                           |
      |                                                                                           struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *
sklearn/metrics/_dist_metrics.c:32461:201: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_SEuclideanDistance64 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *’
32461 | ance64_rdist_csr(struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_SEuclideanDistance64 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x1_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x1_indices, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x2_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x2_indices, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_end, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_end, CYTHON_UNUSED __pyx_t_7sklearn_5utils_9_typedefs_intp_t const __pyx_v_size) {
      |                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19MinkowskiDistance64_dist’:
sklearn/metrics/_dist_metrics.c:34594:86: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19MinkowskiDistance64_rdist’ from incompatible pointer type [-Wincompatible-pointer-types]
34594 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_19MinkowskiDistance64_rdist(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *)__pyx_v_self), __pyx_v_x1, __pyx_v_x2, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float64_t)-1.0))) __PYX_ERR(1, 1457, __pyx_L1_error)
      |                                                                                     ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                      |
      |                                                                                      struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *
sklearn/metrics/_dist_metrics.c:34451:195: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MinkowskiDistance64 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *’
34451 | iDistance64_rdist(struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MinkowskiDistance64 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x1, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x2, __pyx_t_7sklearn_5utils_9_typedefs_intp_t __pyx_v_size) {
      |                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19MinkowskiDistance64_dist_csr’:
sklearn/metrics/_dist_metrics.c:35609:90: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19MinkowskiDistance64_rdist_csr’ from incompatible pointer type [-Wincompatible-pointer-types]
35609 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_19MinkowskiDistance64_rdist_csr(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *)__pyx_v_self), __pyx_v_x1_data, __pyx_v_x1_indices, __pyx_v_x2_data, __pyx_v_x2_indices, __pyx_v_x1_start, __pyx_v_x1_end, __pyx_v_x2_start, __pyx_v_x2_end, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float64_t)-1.0))) __PYX_ERR(1, 1564, __pyx_L1_error)
      |                                                                                         ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                          |
      |                                                                                          struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *
sklearn/metrics/_dist_metrics.c:34998:199: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MinkowskiDistance64 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *’
34998 | tance64_rdist_csr(struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MinkowskiDistance64 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x1_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x1_indices, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x2_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x2_indices, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_end, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_end, CYTHON_UNUSED __pyx_t_7sklearn_5utils_9_typedefs_intp_t const __pyx_v_size) {
      |                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_21MahalanobisDistance64_dist’:
sklearn/metrics/_dist_metrics.c:36611:88: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_21MahalanobisDistance64_rdist’ from incompatible pointer type [-Wincompatible-pointer-types]
36611 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_21MahalanobisDistance64_rdist(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *)__pyx_v_self), __pyx_v_x1, __pyx_v_x2, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float64_t)-1.0))) __PYX_ERR(1, 1649, __pyx_L1_error)
      |                                                                                       ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                        |
      |                                                                                        struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *
sklearn/metrics/_dist_metrics.c:36458:199: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MahalanobisDistance64 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *’
36458 | istance64_rdist(struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MahalanobisDistance64 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x1, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x2, __pyx_t_7sklearn_5utils_9_typedefs_intp_t __pyx_v_size) {
      |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_21MahalanobisDistance64_dist_csr’:
sklearn/metrics/_dist_metrics.c:37439:92: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_21MahalanobisDistance64_rdist_csr’ from incompatible pointer type [-Wincompatible-pointer-types]
37439 |  __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_21MahalanobisDistance64_rdist_csr(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *)__pyx_v_self), __pyx_v_x1_data, __pyx_v_x1_indices, __pyx_v_x2_data, __pyx_v_x2_indices, __pyx_v_x1_start, __pyx_v_x1_end, __pyx_v_x2_start, __pyx_v_x2_end, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float64_t)-1.0))) __PYX_ERR(1, 1730, __pyx_L1_error)
      |                                                                                          ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                           |
      |                                                                                           struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *

sklearn/metrics/_dist_metrics.c:37035:203: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MahalanobisDistance64 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *’
37035 | nce64_rdist_csr(struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MahalanobisDistance64 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x1_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x1_indices, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x2_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x2_indices, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_end, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_end, __pyx_t_7sklearn_5utils_9_typedefs_intp_t const __pyx_v_size) {
      |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19HaversineDistance64_dist’:
sklearn/metrics/_dist_metrics.c:42467:86: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19HaversineDistance64_rdist’ from incompatible pointer type [-Wincompatible-pointer-types]
42467 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_19HaversineDistance64_rdist(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *)__pyx_v_self), __pyx_v_x1, __pyx_v_x2, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float64_t)-1.0))) __PYX_ERR(1, 2635, __pyx_L1_error)
      |                                                                                     ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                      |
      |                                                                                      struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *
sklearn/metrics/_dist_metrics.c:42396:209: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_HaversineDistance64 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *’
42396 | ist(CYTHON_UNUSED struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_HaversineDistance64 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x1, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x2, CYTHON_UNUSED __pyx_t_7sklearn_5utils_9_typedefs_intp_t __pyx_v_size) {
      |                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19HaversineDistance64_dist_csr’:
sklearn/metrics/_dist_metrics.c:43002:90: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19HaversineDistance64_rdist_csr’ from incompatible pointer type [-Wincompatible-pointer-types]
43002 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_19HaversineDistance64_rdist_csr(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *)__pyx_v_self), __pyx_v_x1_data, __pyx_v_x1_indices, __pyx_v_x2_data, __pyx_v_x2_indices, __pyx_v_x1_start, __pyx_v_x1_end, __pyx_v_x2_start, __pyx_v_x2_end, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float64_t)-1.0))) __PYX_ERR(1, 2663, __pyx_L1_error)
      |                                                                                         ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                          |
      |                                                                                          struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *
sklearn/metrics/_dist_metrics.c:4493:213: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_HaversineDistance64 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric64 *’
 4493 | csr(CYTHON_UNUSED struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_HaversineDistance64 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x1_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x1_indices, __pyx_t_7sklearn_5utils_9_typedefs_float64_t const *__pyx_v_x2_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x2_indices, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_end, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_end, CYTHON_UNUSED __pyx_t_7sklearn_5utils_9_typedefs_intp_t const __pyx_v_size); /* proto*/
      |                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_pf_7sklearn_7metrics_13_dist_metrics_16DistanceMetric32_22_pairwise_sparse_dense’:
sklearn/metrics/_dist_metrics.c:48708:29: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
48708 |             __pyx_v_x2_data = ((&(*((__pyx_t_7sklearn_5utils_9_typedefs_float32_t const  *) ( /* dim=1 */ ((char *) (((__pyx_t_7sklearn_5utils_9_typedefs_float32_t const  *) ( /* dim=0 */ (__pyx_v_Y_data.data + __pyx_t_18 * __pyx_v_Y_data.strides[0]) )) + __pyx_t_19)) )))) + (__pyx_v_i2 * __pyx_v_n_features));
      |                             ^
sklearn/metrics/_dist_metrics.c: In function ‘__pyx_pf_7sklearn_7metrics_13_dist_metrics_16DistanceMetric32_24_pairwise_dense_sparse’:
sklearn/metrics/_dist_metrics.c:49493:27: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
49493 |           __pyx_v_x1_data = ((&(*((__pyx_t_7sklearn_5utils_9_typedefs_float32_t const  *) ( /* dim=1 */ ((char *) (((__pyx_t_7sklearn_5utils_9_typedefs_float32_t const  *) ( /* dim=0 */ (__pyx_v_X_data.data + __pyx_t_15 * __pyx_v_X_data.strides[0]) )) + __pyx_t_16)) )))) + (__pyx_v_i1 * __pyx_v_n_features));
      |                           ^
sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19EuclideanDistance32_dist_csr’:
sklearn/metrics/_dist_metrics.c:51154:90: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19EuclideanDistance32_rdist_csr’ from incompatible pointer type [-Wincompatible-pointer-types]
51154 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_19EuclideanDistance32_rdist_csr(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *)__pyx_v_self), __pyx_v_x1_data, __pyx_v_x1_indices, __pyx_v_x2_data, __pyx_v_x2_indices, __pyx_v_x1_start, __pyx_v_x1_end, __pyx_v_x2_start, __pyx_v_x2_end, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float32_t)-1.0))) __PYX_ERR(1, 3657, __pyx_L1_error)
      |                                                                                         ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                          |
      |                                                                                          struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *
sklearn/metrics/_dist_metrics.c:50787:213: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_EuclideanDistance32 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *’
50787 | csr(CYTHON_UNUSED struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_EuclideanDistance32 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x1_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x1_indices, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x2_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x2_indices, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_end, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_end, CYTHON_UNUSED __pyx_t_7sklearn_5utils_9_typedefs_intp_t const __pyx_v_size) {
      |                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_20SEuclideanDistance32_dist’:
sklearn/metrics/_dist_metrics.c:51659:87: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_20SEuclideanDistance32_rdist’ from incompatible pointer type [-Wincompatible-pointer-types]
51659 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_20SEuclideanDistance32_rdist(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *)__pyx_v_self), __pyx_v_x1, __pyx_v_x2, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float32_t)-1.0))) __PYX_ERR(1, 3706, __pyx_L1_error)
      |                                                                                      ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                       |
      |                                                                                       struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *
sklearn/metrics/_dist_metrics.c:51560:197: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_SEuclideanDistance32 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *’
51560 | Distance32_rdist(struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_SEuclideanDistance32 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x1, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x2, __pyx_t_7sklearn_5utils_9_typedefs_intp_t __pyx_v_size) {
      |                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_20SEuclideanDistance32_dist_csr’:
sklearn/metrics/_dist_metrics.c:52474:91: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_20SEuclideanDistance32_rdist_csr’ from incompatible pointer type [-Wincompatible-pointer-types]
52474 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_20SEuclideanDistance32_rdist_csr(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *)__pyx_v_self), __pyx_v_x1_data, __pyx_v_x1_indices, __pyx_v_x2_data, __pyx_v_x2_indices, __pyx_v_x1_start, __pyx_v_x1_end, __pyx_v_x2_start, __pyx_v_x2_end, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float32_t)-1.0))) __PYX_ERR(1, 3786, __pyx_L1_error)
      |                                                                                          ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                           |
      |                                                                                           struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *
sklearn/metrics/_dist_metrics.c:52083:201: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_SEuclideanDistance32 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *’
52083 | ance32_rdist_csr(struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_SEuclideanDistance32 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x1_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x1_indices, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x2_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x2_indices, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_end, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_end, CYTHON_UNUSED __pyx_t_7sklearn_5utils_9_typedefs_intp_t const __pyx_v_size) {
      |                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19MinkowskiDistance32_dist’:
sklearn/metrics/_dist_metrics.c:54216:86: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19MinkowskiDistance32_rdist’ from incompatible pointer type [-Wincompatible-pointer-types]
54216 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_19MinkowskiDistance32_rdist(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *)__pyx_v_self), __pyx_v_x1, __pyx_v_x2, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float32_t)-1.0))) __PYX_ERR(1, 4037, __pyx_L1_error)
      |                                                                                     ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                      |
      |                                                                                      struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *
sklearn/metrics/_dist_metrics.c:54073:195: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MinkowskiDistance32 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *’
54073 | iDistance32_rdist(struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MinkowskiDistance32 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x1, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x2, __pyx_t_7sklearn_5utils_9_typedefs_intp_t __pyx_v_size) {
      |                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19MinkowskiDistance32_dist_csr’:
sklearn/metrics/_dist_metrics.c:55231:90: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19MinkowskiDistance32_rdist_csr’ from incompatible pointer type [-Wincompatible-pointer-types]
55231 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_19MinkowskiDistance32_rdist_csr(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *)__pyx_v_self), __pyx_v_x1_data, __pyx_v_x1_indices, __pyx_v_x2_data, __pyx_v_x2_indices, __pyx_v_x1_start, __pyx_v_x1_end, __pyx_v_x2_start, __pyx_v_x2_end, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float32_t)-1.0))) __PYX_ERR(1, 4144, __pyx_L1_error)
      |                                                                                         ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                          |
      |                                                                                          struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *
sklearn/metrics/_dist_metrics.c:54620:199: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MinkowskiDistance32 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *’
54620 | tance32_rdist_csr(struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MinkowskiDistance32 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x1_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x1_indices, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x2_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x2_indices, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_end, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_end, CYTHON_UNUSED __pyx_t_7sklearn_5utils_9_typedefs_intp_t const __pyx_v_size) {
      |                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_21MahalanobisDistance32_dist’:
sklearn/metrics/_dist_metrics.c:56233:88: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_21MahalanobisDistance32_rdist’ from incompatible pointer type [-Wincompatible-pointer-types]
56233 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_21MahalanobisDistance32_rdist(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *)__pyx_v_self), __pyx_v_x1, __pyx_v_x2, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float32_t)-1.0))) __PYX_ERR(1, 4229, __pyx_L1_error)
      |                                                                                       ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                        |
      |                                                                                        struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *
sklearn/metrics/_dist_metrics.c:56080:199: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MahalanobisDistance32 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *’
56080 | istance32_rdist(struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MahalanobisDistance32 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x1, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x2, __pyx_t_7sklearn_5utils_9_typedefs_intp_t __pyx_v_size) {
      |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_21MahalanobisDistance32_dist_csr’:
sklearn/metrics/_dist_metrics.c:57061:92: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_21MahalanobisDistance32_rdist_csr’ from incompatible pointer type [-Wincompatible-pointer-types]
57061 |  __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_21MahalanobisDistance32_rdist_csr(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *)__pyx_v_self), __pyx_v_x1_data, __pyx_v_x1_indices, __pyx_v_x2_data, __pyx_v_x2_indices, __pyx_v_x1_start, __pyx_v_x1_end, __pyx_v_x2_start, __pyx_v_x2_end, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float32_t)-1.0))) __PYX_ERR(1, 4310, __pyx_L1_error)
      |                                                                                          ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                           |
      |                                                                                           struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *

sklearn/metrics/_dist_metrics.c:56657:203: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MahalanobisDistance32 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *’
56657 | nce32_rdist_csr(struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_MahalanobisDistance32 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x1_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x1_indices, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x2_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x2_indices, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_end, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_end, __pyx_t_7sklearn_5utils_9_typedefs_intp_t const __pyx_v_size) {
      |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19HaversineDistance32_dist’:
sklearn/metrics/_dist_metrics.c:62089:86: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19HaversineDistance32_rdist’ from incompatible pointer type [-Wincompatible-pointer-types]
62089 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_19HaversineDistance32_rdist(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *)__pyx_v_self), __pyx_v_x1, __pyx_v_x2, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float32_t)-1.0))) __PYX_ERR(1, 5215, __pyx_L1_error)
      |                                                                                     ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                      |
      |                                                                                      struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *
sklearn/metrics/_dist_metrics.c:62018:209: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_HaversineDistance32 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *’
62018 | ist(CYTHON_UNUSED struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_HaversineDistance32 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x1, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x2, CYTHON_UNUSED __pyx_t_7sklearn_5utils_9_typedefs_intp_t __pyx_v_size) {
      |                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

sklearn/metrics/_dist_metrics.c: In function ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19HaversineDistance32_dist_csr’:
sklearn/metrics/_dist_metrics.c:62624:90: error: passing argument 1 of ‘__pyx_f_7sklearn_7metrics_13_dist_metrics_19HaversineDistance32_rdist_csr’ from incompatible pointer type [-Wincompatible-pointer-types]
62624 |   __pyx_t_1 = __pyx_f_7sklearn_7metrics_13_dist_metrics_19HaversineDistance32_rdist_csr(((struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *)__pyx_v_self), __pyx_v_x1_data, __pyx_v_x1_indices, __pyx_v_x2_data, __pyx_v_x2_indices, __pyx_v_x1_start, __pyx_v_x1_end, __pyx_v_x2_start, __pyx_v_x2_end, __pyx_v_size); if (unlikely(__pyx_t_1 == ((__pyx_t_7sklearn_5utils_9_typedefs_float32_t)-1.0))) __PYX_ERR(1, 5243, __pyx_L1_error)
      |                                                                                         ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                                                          |
      |                                                                                          struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *
sklearn/metrics/_dist_metrics.c:4561:213: note: expected ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_HaversineDistance32 *’ but argument is of type ‘struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_DistanceMetric32 *’
 4561 | csr(CYTHON_UNUSED struct __pyx_obj_7sklearn_7metrics_13_dist_metrics_HaversineDistance32 *__pyx_v_self, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x1_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x1_indices, __pyx_t_7sklearn_5utils_9_typedefs_float32_t const *__pyx_v_x2_data, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const *__pyx_v_x2_indices, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x1_end, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_start, __pyx_t_7sklearn_5utils_9_typedefs_int32_t const __pyx_v_x2_end, CYTHON_UNUSED __pyx_t_7sklearn_5utils_9_typedefs_intp_t const __pyx_v_size); /* proto*/
      |                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~

error: command '/usr/bin/gcc' failed with exit code 1
```

### Steps/Code to Reproduce

```
sudo -E podman run -ti --rm fedora:rawhide
yum update -y
yum install -y git gcc gcc-c++ python3-devel python3-pip python3-setuptools python3-wheel python3-Cython python3-numpy python3-scipy python3-joblib python3-threadpoolctl
git clone https://github.com/scikit-learn/scikit-learn.git
cd scikit-learn
git checkout 1.4.1.post1
python3 setup.py build
```

Fixes (see https://github.com/scikit-learn/scikit-learn/issues/28530#issuecomment-1964479520):
```
sudo -E podman run -ti --rm fedora:rawhide
yum update -y
yum install -y git gcc gcc-c++ python3-devel python3-pip python3-setuptools python3-wheel python3-Cython python3-numpy python3-scipy python3-joblib python3-threadpoolctl
git clone https://github.com/scikit-learn/scikit-learn.git
cd scikit-learn
git checkout main
find sklearn/metrics/_dist_metrics.pyx.tp -type f | xargs sed -i 's/cdef inline {{INPUT_DTYPE_t}} rdist/cdef {{INPUT_DTYPE_t}} rdist/g'
python3 setup.py build
```

### Expected Results

Build successful.

### Actual Results

As shown above.

### Versions

```shell
1.4.1.post1
```

user: hswong3i
labels: Needs Reproducible Code: Issue requires reproducible code
issue_28528: state: open
title: Update cluster metrics to handle edge cases.
body: #### Reference Issues/PRs

Closes https://github.com/scikit-learn/scikit-learn/issues/15717. Supersedes https://github.com/scikit-learn/scikit-learn/pull/15815.

#### What does this implement/fix? Explain your changes.

Modify  `silhouette_samples`, `silhouette_score`, `calinski_harabasz_score`, and `davies_bouldin_score` to be compatible with `GridSearchCV()` when the numbers of clusters is equal to 1 or equal to n_samples.

#### Any other comments?

None.

cc @jnothman 
user: fkdosilovic
issue_28521: state: open
title: FIX safe indexing for polars `Series`
body: Towards #28488.

The initial goal of this PR is to make `_safe_indexing` work for polars Series and changing `_is_polars_df` into `_is_polars_df_or_series` suffices.

However when extending the tests for pandas Series and DataFrame to polars, I found some other places that may need to be fixed (e.g., `_polars_indexing`).
user: Charlie-XIAO
issue_28520: state: open
title: Create estimators for inference only
body: ### Describe the workflow you want to enable

Allow a trained estimator to be converted into a form suitable only for predict/transform type operations and not fitting. In many cases, the estimator could be made more compact or performant as part of this transformation. 

For instance, feature selection steps in a pipeline may rely on complex models during training, but at inference they simply drop unused features. When deploying the model, conversion of the feature selection step to a simpler form could save memory and model load time. 

### Describe your proposed solution

Add a new method to BaseEstimator `prep_for_inference(self)` which returns a model which retains all predict/transform methods but does not necessarily support fitting. By default it would return `self`. Estimators and transformers could override this as necessary. Pipeline would convert each step. 

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: gorj-tessella
labels: Needs Triage: Issue requires triage
issue_28519: state: open
title: ENH add shuffle to GroupKFold
body: #### Reference Issues/PRs
#20520


#### What does this implement/fix? Explain your changes.
This update introduces a shuffle feature to `GroupKFold`, along with a new class, `RepeatedGroupKFold`, which supports repeating `GroupKFold` n times. When `shuffle=False`, the behavior remains unchanged, ensuring that groups are split to achieve folds of as equal size as possible. Setting `shuffle=True` shuffles the unique groups before they are assigned to folds, without prioritizing the size of the individual groups.

#### Any other comments?
Although enabling shuffling does not guarantee that folds will be of similar size, in scenarios where groups are roughly equal in size (a common situation) the size difference between folds should be minimal.
user: zvealey
issue_28517: state: open
title: MNT more informative error message for `UnsetMetadataPassedError`
body: #### What does this implement/fix? Explain your changes.

This PR aims to improve the information provided when `UnsetMetadataPassedError` is raised.

The error message will now explicitly state, that the method they are asked to set a metadata request is called from which method in the meta-estimator object.

#### More detailed explanation

This is especially important when a method in the router is routing to several methods in a consumer. See  #28261 for `RANSACRegressor` as an example, where `RANSACRegressor.fit()` routes to both `estimator.fit()` and `estimator.score()`. The user will receive the following error:

`
sklearn.exceptions.UnsetMetadataPassedError: [sample_weight] are passed but are not explicitly set as requested or not requested for LinearRegression.score, which is used within RANSACRegressor.fit. Call LinearRegression.set_score_request({metadata}=True) for each metadata.
`

when they have done `LinearRegression.set_fit_request(sample_weight=True)`, but where not aware that they need to do `set_score_request(sample_weight=True)` as well. At the moment, the error message they would receive in such a case would confuse many people.

Additionally, in case of composed methods like `fit_transform` and `fit_predict` the message is now improved to suggest to use `set_fit_request`, instead of setting the request for the composed method (which would result in an error).

It also works over several layers.

@adrinjalali :)
user: StefanieSenger
issue_28513: state: open
title: FIX `permutation_importance` with polars dataframe raises warning on feature names
body: #### Reference Issues/PRs

Towards #28488.

#### What does this implement/fix? Explain your changes.

In `permutation_importance` we forcefully convert polars dataframes to arrays, causing `UserWarning: X does not have valid feature names, but XXX was fitted with feature names`.

I added no extra tests, except for extending all tests on `pandas` to also include `polars`.

user: Charlie-XIAO
issue_28512: state: open
title: DOC [PST] disable gallery link tweaks when `html-noplot` to avoid warnings
body: **Please note that this PR targets the `new_web_theme` branch!**

Towards #28084. For context see https://github.com/scikit-learn/scikit-learn/pull/28331#issuecomment-1959582859. In short, when built without running gallery examples (`make html-noplot`) the badge links and download links could be missing; we do not want to tweak them then. This PR also improves the warning messages a bit.

@ogrisel
user: Charlie-XIAO
issue_28509: state: open
title: FIX matthews_corrcoef returns zero for perfect prediction
body: #### Reference Issues/PRs
Fixes #25258
See also #19977

#### What does this implement/fix? Explain your changes.
The Matthews correlation coefficient is ill-defined (due to zero division) when only one class is present in either the true or predicted labels.

If only one of the true or predicted labels contains a single class, the limit value is 0 (can be shown using polar coordinates). This is sensible as it suggests that the model either provided constant predictions on non-constant data, or variable predictions on single-class data. In such cases, the metric should return a value of 0.

However, if both the true and predicted labels contain only a single class, the limit does not exist, rendering the metric undefined. Also, if both true and model labels are single class, the model succeeded in trivial task and we genuinely can't tell  if the correlation is good or poor. Consequently, in this scenario, the metric should return a nan value. This behavior was chosen to avoid returning 0 for perfect predictions on single-class data.

#### Any other comments?

I noticed that for some other metrics the handling of zero division was based on `_prf_divide()`, but it didn't seem a good fit for this case as its default behavior is to return 0 on both cases and IMHO directly returning nan or 0 yields more readable code and is simpler for users. 

@marctorsoc @glemaitre WDYT?
user: Redjest
issue_28507: state: open
title: Allow `RandomForest*` and `ExtraTrees*` to have a higher max_samples than 1.0 when `bootstrap=True`
body: ### Describe the workflow you want to enable

Currently, random/extra forests can bootstrap sample the data such that `max_samples \in (0.0, 1.0]`. This enables an out-of-bag sample estimate in forests.

However, this only allows you to sample in principle up to at most 63% unique samples and then 37% of unique samples are for out-of-bag estimation. However, you should be able to control this parameter to a proportion greater. For instance, perhaps I want to leverage 80% of my data to fit each tree, and 20% to estimate oob performance. This requires one to set `max_samples=1.6`. 

Beyond that, no paper suggests that 63% is required cutoff for bootstrapping the samples in Random/Extra forest. I am happy to submit a PR if the core-dev team thinks the propose solution is simple and reasonable.

See https://stats.stackexchange.com/questions/126107/expected-proportion-of-the-sample-when-bootstrapping for a good reference and explanation.

### Describe your proposed solution

The proposed solution is actually backwards-compatable and adds minimal complexity to the codebase.

1. We change https://github.com/scikit-learn/scikit-learn/blob/38c8cc3bab151b76ed890a4b690871e0fa404426/sklearn/ensemble/_forest.py#L95-L125 to the following LOC:
2.
```Python

def _get_n_samples_bootstrap(n_samples, max_samples):
    """
    Get the number of samples in a bootstrap sample.

    The expected total number of unique samples in a bootstrap sample is
    required to be at most ``n_samples - 1``.
    This is equivalent to the expected number of out-of-bag samples being at
    least 1.

    Parameters
    ----------
    n_samples : int
        Number of samples in the dataset.
    max_samples : int or float
        The maximum number of samples to draw from the total available:
            - if float, this indicates a fraction of the total;
            - if int, this indicates the exact number of samples;
            - if None, this indicates the total number of samples.

    Returns
    -------
    n_samples_bootstrap : int
        The total number of samples to draw for the bootstrap sample.
    """
    if max_samples is None:
        return n_samples

    if isinstance(max_samples, Integral):
        expected_oob_samples = (1 - np.exp(-max_samples / n_samples)) * n_samples
        if expected_oob_samples >= n_samples - 1:
            raise ValueError(
                "The expected number of unique samples in the bootstrap sample"
                f" must be at most {n_samples - 1}. It is: {expected_oob_samples}"
            )
        return max_samples

    if isinstance(max_samples, Real):
        expected_oob_samples = (1 - np.exp(-max_samples)) * n_samples
        if expected_oob_samples >= n_samples - 1:
            raise ValueError(
                "The expected number of unique samples in the bootstrap sample"
                f" must be at most {n_samples - 1}. It is: {expected_oob_samples}"
            )
        return max(round(n_samples * max_samples), 1)
```

Note, we probably want some reasonable control over how large `max_samples` can be relative to `n_samples`. For instance if `max_samples = 10*n_samples`, this results in you pretty much sampling all unique samples per tree and almost no samples for oob computation. Thus a reasonable cutoff is we always allow at least 1 sample to be oob.

- if `max_samples` is an integer -> then it must be `(1 - e^(-max_samples/n_samples)) * n_samples > 1`
- if `max_samples` is a float -> then it must be that `(1 - e^(-max_samples)) * n_samples > 1` (i.e. you are expected to sample at least 1 sample out of bag). 

Alternatively, we can impose a reasonable heuristic of 5 samples. I think regardless it works for most use-cases because people would typically want to change the in-bag percentage from 63% to say 80% or 90% at most, but not 99.99%

### Describe alternatives you've considered, if relevant

There is no other way of allowing this functionality without forking the code.

### Additional context

This allows flexibility in terms of the trees and may help in supporting other issues that require more fine-grained control over what is in-bag vs oob such as #19710.
user: adam2392
labels: Needs Triage: Issue requires triage
issue_28506: state: open
title: BUILD/CI Switch to Meson as main build backend
body: Opening this PR to get some feed-back about people feelings on making Meson our main build backend. Personnally I have used this daily for my scikit-learn and and I have not noticed any real blocker issues.

I kept a single build with setuptools: `pymin_conda_defaults_openblas`. No particular reason why I chose it.

It could potentially be an option to do a separate doc PR. Two reasons for this:
- the doc needs a bit of work to mention the meson editable command. In particular we can remove `--no-use-pep517` everywhere
- the old pip command will still work and will ignore `pyproject.toml` (thanks to the `--no-use-pep517`)

Some questions I have about the doc:
- there are plenty of places where we can remove `--no-use-pep517` :tada: 
- As to the recommended way to build, I am slightly leaning towards giving the editable command in verbose mode (`pip install --verbose --no-build-isolation --editable . --config-settings editable-verbose=true`). It is quite long but you have to run it once and then `import sklearn` will recompile as needed. I find having some feed-back when things recompile very reassuring (meson did recompile things and something is happening this is why my script has not started running yet). This new behaviour probably needs to be explained in the doc ...
- I think using [spin](https://github.com/scientific-python/spin) may be an option in the medium-term. With spin, the workflow is more similar to the current one: `spin build` when you modify a Cython file, `spin test` to test. So changing all the doc one way to explain the new way with meson editable and then in a month or so revert and explain spin may not be a good use of my time. The big advantage of `spin` in my mind is simplicity of the command and the fact that it is discoverable on the command-line. You don't have to look at the doc or your shell history each time you need to find the very long-winded command you need to type.

Full-disclosure there are two nice-to-have features from meson-python but I don't consider them a real blocker for daily work:
- https://github.com/mesonbuild/meson-python/pull/569. This is the reason there is `pytest<8` in this PR (note the bug only happens with `pytest --pyargs` which is used in the CI but I don't think in day-to-day work?). We also skip one test on Meson because of this right now. Hopefully this will be merged and a 0.16.0 will be released in the not-to-far future.
- https://github.com/mesonbuild/meson-python/pull/579. More a convenience thing about the message that gets printed when recompilation or no recompilation happens.
user: lesteve
issue_28505: state: open
title: [WIP] [POC] FEA public objective function methods in estimators
body: towards #28169

Not sure where to put the `ObjectiveFunction` dataclass. I put it in sklearn.base for now but open to suggestions.
The design is not final. This PR is there to help discussing the design in the linked issue #28169 where discussion regarding the general design of this API should be discussed.
user: jeremiedbb
issue_28503: state: open
title: DOC add list of regressors that support multioutput regression natively
body: #### Reference Issues/PRs
Fixes #21484

#### What does this implement/fix? Explain your changes.
Includes a list of multioutput regressors.

#### Any other comments?
user: rprkh
issue_28496: state: open
title: Remove the float64 accumulator in r2_score sums
body: #2158 was merged without non regression tests 10 years ago to use `float64` accumulators in the 2 sum operations (across samples) used to compute the numerator and denominator of the final expression computed by `r2_score`.

Since the summed terms are all positive, [I have the feeling](https://github.com/scikit-learn/scikit-learn/pull/27904#discussion_r1494858270) that those sums should be numerically stable by default (contrary to what was suggested in #2158). I tried to conduct an empirical study both on `main` and on this PR to see if I could trigger non-numerically stable sums in r2 scores computed on 100,000,000 predictions.

Here is the code to collect the results:

<details>

```python
# %%
from sklearn.metrics import r2_score
import numpy as np
import os
import subprocess
import pandas as pd
import sklearn


n_samples = int(1e8)
rng = np.random.default_rng(0)
cwd = os.getcwd()
os.chdir(os.path.dirname(sklearn.__file__))
branch_name = (
    subprocess.check_output("git rev-parse --abbrev-ref HEAD".split())
    .strip()
    .decode("utf-8")
)
# check that there are no uncommitted changes:
assert not subprocess.check_output("git diff".split()).strip()
os.chdir(cwd)

# Precompute RNG numbers to be assembled later at each iteration:
base_signal = rng.normal(loc=0, scale=1, size=n_samples)
base_noise = rng.normal(loc=0, scale=1, size=n_samples)
w = rng.uniform(0, 1, size=n_samples)

signal_buffer = np.empty_like(base_signal)
noise_buffer = np.empty_like(base_noise)

print(f"Branch: {branch_name}")
records = []
for loc in np.logspace(-3, 3, 3):
    for scale in np.logspace(-6, 6, 5):
        for noise_scale_factor in [1e-6, 0.1, 0.5, 0.999]:
            noise_scale = scale * noise_scale_factor

            # Generate the data at single precision
            signal_buffer[:] = base_signal
            signal_buffer *= scale
            signal_buffer += loc
            y_true = signal_buffer.astype(np.float32, copy=True)

            noise_buffer[:] = base_noise
            noise_buffer *= noise_scale
            noise_buffer += signal_buffer
            y_pred = noise_buffer.astype(np.float32, copy=True)
            sample_weight = w.astype(np.float32, copy=True)

            # Peform the computation at single precision (optionally with
            # double precision upcasts depending on the code in scikit-learn):
            r2_f32 = r2_score(y_true, y_pred, sample_weight=sample_weight)

            # Upcast the same data to perform all computation with double precision.
            r2_f64 = r2_score(
                y_true.astype(np.float64),
                y_pred.astype(np.float64),
                sample_weight=sample_weight.astype(np.float64),
            )
            abs_diff = np.abs(r2_f32 - r2_f64)
            print(
                f"loc={loc:.2e}, scale={scale:.2e}, noise_scale_factor={noise_scale_factor:.2e}, "
                f"score_float32_input={r2_f32:.6f}, score_float64_input={r2_f64:.6f}, "
                f"abs_diff={abs_diff:.6f}"
            )
            records.append(
                {
                    "loc": loc,
                    "scale": scale,
                    "noise_scale_factor": noise_scale_factor,
                    "r2_float32": r2_f32,
                    "r2_float64": r2_f64,
                }
            )

# %%
filename = f"float32_r2_score_{branch_name}.parquet"
pd.DataFrame(records).to_parquet(filename)
```

</details>

Note that this code tries many `y_true` scales and offsets and correlations with `y_pred`.

I ran it both on this branch and on `main` to evaluate the impact of the `np.float64` upcast in the sum operations.

Here is the code to combine the results:

<details>

```python
import pandas as pd
import numpy as np

df_main = pd.read_parquet("float32_r2_score_main.parquet")
df_branch = pd.read_parquet("float32_r2_score_float32-r2_score.parquet")

df = pd.merge(
    df_main,
    df_branch,
    on=("loc", "scale", "noise_scale_factor"),
    suffixes=("_main", "_branch"),
)
np.testing.assert_allclose(df["r2_float64_main"], df["r2_float64_branch"])

df["diff_float32_branches"] = np.abs(df["r2_float32_main"] - df["r2_float32_branch"])
df["diff_float32_float64_main"] = np.abs(df["r2_float32_main"] - df["r2_float64_main"])
```

</details>

- First we can observe that when computing from the difference in r2 score compute between the 2 branches on float32 data, the difference is never bigger than 5e-06 which seems ok to me:

```python
print(df.sort_values("diff_float32_branches", ascending=False).head(5).to_markdown())
```

|    |      loc |   scale |   noise_scale_factor |   r2_float32_main |   r2_float64_main |   r2_float32_branch |   r2_float64_branch |   diff_float32_branches |
|---:|---------:|--------:|---------------------:|------------------:|------------------:|--------------------:|--------------------:|------------------------:|
|  3 |    0.001 |   1e-06 |                0.999 |        0.00166918 |       0.00166847  |          0.00166547 |         0.00166847  |             3.71002e-06 |
| 23 |    1     |   1e-06 |                0.999 |        0.00447393 |       0.000944231 |          0.00447714 |         0.000944231 |             3.20946e-06 |
| 39 |    1     |   1e+06 |                0.999 |        0.00166847 |       0.00166847  |          0.00166625 |         0.00166847  |             2.22143e-06 |
| 19 |    0.001 |   1e+06 |                0.999 |        0.00166847 |       0.00166847  |          0.00166625 |         0.00166847  |             2.22143e-06 |
| 59 | 1000     |   1e+06 |                0.999 |        0.00166847 |       0.00166847  |          0.00166625 |         0.00166847  |             2.22143e-06 


- However, when comparing r2_score with `float32` data and their `float64` counter part, both on `main`, there are a few edge cases where it's not numerically stable:

```python
print(df.sort_values("diff_float32_float64_main", ascending=False).head(5).to_markdown())
```

|    |   loc |   scale |   noise_scale_factor |   r2_float32_main |   r2_float64_main |   r2_float32_branch |   r2_float64_branch |   diff_float32_float64_main |
|---:|------:|--------:|---------------------:|------------------:|------------------:|--------------------:|--------------------:|----------------------------:|
| 47 |  1000 |   0.001 |                0.999 |        0.770511   |       0.00136753  |          0.770511   |         0.00136753  |                 0.769143    |
| 46 |  1000 |   0.001 |                0.5   |        0.942406   |       0.749375    |          0.942406   |         0.749375    |                 0.19303     |
| 45 |  1000 |   0.001 |                0.1   |        0.99756    |       0.98938     |          0.99756    |         0.98938     |                 0.00817938  |
| 23 |     1 |   1e-06 |                0.999 |        0.00447393 |       0.000944231 |          0.00447714 |         0.000944231 |                 0.0035297   |
| 22 |     1 |   1e-06 |                0.5   |        0.749516   |       0.748628    |          0.749518   |         0.748628    |                 0.000888106 |

But note that this numerical stability problems are kind of expected since the `y_true` has a large offset (around 1000) and a very small scale of variation (0.001). So those are pathological data to represent in `float32`.

More importantly, this problem already exist on `main` and this PR where I disable the `np.float64` does not change anything in that respect.

So my conclusions is that:

- I could not find cases where the `np.float64` in the sum operations are significantly improving numerical stability.
- Pathological cases that are numerically unstable when computing `r2_score` on `np.float32` data be fixed by only using `np.float64` accumulators in the sum: all the computation should be done in `float64`.
user: ogrisel
issue_28494: state: open
title: FEA Metadata routing for `SelfTrainingClassifier`
body: #### Reference Issues/PRs
Towards: #22893 


#### What does this implement/fix? Explain your changes.
Implements metadata routing for `SelfTrainingClassifier`. The unit-tests need to support scoring, decision_function, and prediction, so I am leaning towards adding this as a unit-test possibly with something like Pipeline?

Or seeing if it can fit inside the existing unit-testing framework w/ the other classifiers like BaggingClassifier.


#### Any other comments?
cc: @adrinjalali 

Some open questions:

1. I presume, we want to forward metadata within all the functions possibly?
2. As a result, I'm not sure if the unit-testing approach is the best, so I was wondering if you have any suggestions? Should I try potentially refactoring the existing unit-testing code to allow testing for more than just `fit`?
user: adam2392
issue_28492: state: open
title: Include a lower bound attribute of BaseMixture
body: ### Describe the workflow you want to enable

Currently, there exists a `lower_bound_` attribute in the `fit_predict` method of `BaseMixture`.  However, the entire sequence of lower bounds is not accessible, which makes a convergence analysis more difficult to a user. 

### Describe your proposed solution

In addition to the `lower_bound_` attribute, create a new attribute called `lower_bounds_`, which is a list of floats where each float is a lower bound set in https://github.com/scikit-learn/scikit-learn/blob/92c9b1866fab77412d8fe93cb3716c03d80ad8ed/sklearn/mixture/_base.py#L248.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Creating the list and appending to it would increase memory costs (but not by much). Is this a possible concern?
user: a-wozniakowski
labels: Needs Triage: Issue requires triage
issue_28490: state: open
title: CI Build musllinux wheels
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes #27004
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
I've added a wheel-output for musllinux, for faster installation on (among others) Alpine-Linux.

#### Any other comments?
Since Alpine-Linux is used increasingly, pre-building wheels for Alpine-Linux is a must-have IMO.

I'm unfamiliar with the codebase, so I might've missed something. I tried my best to test it locally by running `CIBW_PROJECT_REQUIRES_PYTHON=">=3.12" CIBW_BUILD=cp312-musllinux_x86_64 cibuildwheel --platform=linux --output-dir wheelhouse --archs x86_64` on my M2 ARM MacBook.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
https://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: kvanzuijlen
issue_28488: state: open
title: Improve "polars" integration (error, warning & linting examples)
body: ### Describe the workflow you want to enable

using polars data (DataFrame, Series) is already supported in many places which is awesome, thank you!!

But in many places there are still
- errors / crashes -> required conversion to numpy/pandas
- warnings -> requires conversion to numpy/pandas
- linting/type problems -> requires updates to typing signalture?

# Examples

## Code

```python
import pandas as pd
import polars as pl
from sklearn.datasets import make_classification
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, cross_validate

X, y = make_classification()

# X = pd.DataFrame(X)
# y = pd.Series(y)

# X = pl.DataFrame(X)
# y = pl.Series(y)

clf = LogisticRegression()

clf.fit(
    X=X,
    y=y,
)

clf.score(
    X=X,  # Lint/Type Problem
    y=y,
)

cross_val_score(
    estimator=clf,
    X=X,  # Lint/Type Problem
    y=y,  # ERROR with polars
)

cross_validate(
    estimator=clf,
    X=X,  # Lint/Type Problem
    y=y,  # ERROR with polars
)

permutation_importance(
    estimator=clf,
    X=X,  # WARNING with polars + Lint/Type Problem
    y=y,
)

clf.predict(X)
```

## Errors / crashes using polars

Both `cross_val_score` and `cross_validate` crash using polars Series with message:
- `TypeError: cannot use `__getitem__` on Series of dtype Int32 with argument (array([18, 21, 22, 23, 24, 25,...`

### Temporary solution

use
- `to_numpy()` -> works with numpy array
- `to_pandas()` -> works with pandas Series

## Warnings

`permutation_importance` creates "UserWarnings" using polars DataFrame with message:
- `UserWarning: X does not have valid feature names, but LogisticRegression was fitted with feature names`

### Temporary Solution

use
- `to_pandas()` -> works with pandas DataFrame

## Linting / Type problems

numpy arrays and pandas DataFrame have "full support" while polars looks a little sad 😆 

![image](https://github.com/scikit-learn/scikit-learn/assets/25177421/b0aa54c2-52e0-4d4c-8c7c-448a9fbb978e)











### Describe your proposed solution

would love to see further support for polars DataFrame / Series types.

Not sure why some things error, because I thought polars provides all the required apis.

Any way to assist / help?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: JulianCologne
labels: Needs Triage: Issue requires triage
issue_28478: state: open
title: DOC add Algolia search bar for improved website search
body: This is a POC using `docsearch` from Algolia that you bring a better UX when it comes on using the search bar of the website.

This PR build on the top of the work of @Charlie-XIAO.

See #6710 for historical context.
user: glemaitre
issue_28469: state: open
title: Suggesting updates on the doc of `sklearn.linear_model.OrthogonalMatchingPursuit`
body: ### Describe the issue linked to the documentation

Hi,

We discover an inconsistency issue between documentation and code in the class [`sklearn.linear_model.OrthogonalMatchingPursuit`](https://scikit-learn.org/dev/modules/generated/sklearn.linear_model.OrthogonalMatchingPursuit.html#sklearn-linear-model-orthogonalmatchingpursuit). As mentioned in the description of parameter `n_nonzero_coefs` and `tol`.

> **n_nonzero_coefs: int, default=None**
Desired number of non-zero entries in the solution. _**If None (by default) this value is set to 10% of n_features.**_

> **tol: float, default=None**
Maximum squared norm of the residual. _**If not None, overrides n_nonzero_coefs.**_

The most relevant piece of source code looks like this:
```
if self.n_nonzero_coefs is None and self.tol is None:
    self.n_nonzero_coefs_ = max(int(0.1 * n_features), 1)
else:
    self.n_nonzero_coefs_ = self.n_nonzero_coefs
```

This piece of code does not logically cover the description in the documentation perfectly. For example, when `n_nonzero_coefs` is None and `tol` is not None, `n_nonzero_coefs` will be still overridden. However as the rule in `n_nonzero_coefs`, `n_nonzero_coefs` should be set to 10% of n_features.

Could you please check it?

### Suggest a potential alternative/fix

Maybe you can reconstruct the if-else branch to cover the situation or update the doc to make it clear.
user: DocInspector
issue_28467: state: open
title: Revisiting meta-routing developer API for defining method mappings
body: For metadata routing, meta-estimators define method mappings through this API:

```python
router = MetadataRouter(...).add(
	 method_mapping=MethodMapping()
     .add(callee="fit", caller="fit")
     .add(callee="transform", caller="fit")
     .add(callee="transform", caller="predict"),
)
```

Currently, we have a shortcut to map every available method dynamically:

```python
MetadataRouter(...).add(method_mapping="one-to-one")
```

And a shortcut to do a single "one-to-one" mapping:

```python
MetadataRouter(...).add(method_mapping="score")
```

https://github.com/scikit-learn/scikit-learn/pull/28422 proposes to remove both theses shortcuts. The proposal makes sense as it forces everything to be explicit and there is only one way to define the method mappings.

On the other hand, I think a majority of the time a meta-estimator does "one-to-one" method mappings. In https://github.com/scikit-learn/scikit-learn/pull/28422#issuecomment-1951377557, I proposed this API:

```python
# Single string for a single mapping:
MethodMapping().add_one_to_one("score")

# Add a second mapping by using a list:
MethodMapping().add_one_to_one(["score", "predict"])
```

I'm mostly considering third party meta-estimator developers and how to make it easier for them to adopt the metadata routing API and define these mappings. If `add_one_to_one` handles a majority of the use cases, I think it's worth introducing even if it a second way to specify the mappings.

user: thomasjpfan
labels: Needs Decision - API: Requires decision regarding API
Metadata Routing: all issues related to metadata routing, slep006, sample props
issue_28464: state: open
title: GroupKFold: Use stable sort to produce consistent splits across installs
body: The use of np.argsort with unstable sorting results in different splits across installations.  This one is hard to write a test for since the splits will always be consistent with multiple runs in the same environment.  Ideas of how to test this would be appreciated, but perhaps it's a simple enough change to motivate without a failing test.
user: marikabergengren
issue_28459: state: open
title: :lock: :robot: CI Update lock files for scipy-dev CI build(s) :lock: :robot:
body: Update lock files.

### Note
If the CI tasks fail, create a new branch based on this PR and add the required fixes to that branch.
user: scikit-learn-bot
issue_28455: state: open
title: [MRG] DOC added examples to docstrings to datasets
body: 
#### Reference Issues/PRs
#27982 


#### What does this implement/fix? Explain your changes.
Added docstring examples to the following functions:
- `make_sparse_coded_signal`
- `make_sparse_uncorrelated`
- `make_swiss_roll`
- `make_s_curve`
user: Cemlyn
issue_28452: state: open
title: Adding a log1p transformer compatible with Pipeline and Grid Search 
body: ### Describe the workflow you want to enable

Using a pipeline and a grid search, i want to check if it is better to pass log1p functions some columns, depending a skew threshold.

The code should go like this  for the pipeline  : 
```python 

pipeline = Pipeline(
    [
        ("log1p", LogColumnTransformer()),
        ("scaler", StandardScaler()),
        ("estimator", RandomForestClassifier()),
    ]
)
````

for the grid_search : 
```python


param_grid = {
    "log1p__threshold": [0.5, 1, 1.5, 3, 3.5],
    "scaler": [StandardScaler(), "passthrough"],
    "estimator__n_estimators": [100, 200, 300],
}
````

and of course : 
``` python
grid = GridSearchCV(
    pipeline,
    param_grid=param_grid,
    cv=5,
    refit=True,
    return_train_score=True,
    n_jobs=-1,
    verbose=0,
)

```

### Describe your proposed solution


I have already implemented such a class and it works.

I Think this is not a sufficient qa code to be intergrated to sklearn but such a feature should be a good idea.

An indicative source code could be found here : [file](https://github.com/AlexandreGazagnes/scikit-transformers/blob/59677a96b737891bb3231da6fe79682e60b2e0db/sktransf/transformer/log.py)

Just as an option, here's the code : 
```python

class LogColumnTransformer(BaseEstimator, TransformerMixin):
    """Logarithm transformer for columns with high skewness"""

    threshold = SkewThreshold()
    ignore_int = Bool()
    force_df_out = Bool()

    def __init__(
        self,
        threshold: int | float = 3,
        ignore_int: bool = False,
        force_df_out: bool = False,
    ) -> None:
        """Init method"""

        if not isinstance(threshold, (float, int)):
            raise TypeError("threshold must be a float or an integer")

        if not isinstance(force_df_out, (int, bool)):
            raise TypeError("out must be a boolean")

        self.force_df_out = force_df_out
        self.ignore_int = ignore_int
        self.threshold = threshold
        self._log_cols = None
        self._standard_cols = None
        self.fitted_columns = None

    def fit(
        self,
        X: pd.DataFrame | np.ndarray | list,
        y: None = None,
    ):
        """Fit method"""

        _X = manage_input(X)
        self.fitted_columns = _X.columns.tolist()

        manage_negatives(_X)

        _X = _X.select_dtypes(include=["number"])

        if self.ignore_int:
            _X = _X.select_dtypes(exclude=["int"])

        # compute skew
        skew = _X.skew().round(3).to_dict()

        self._log_cols = []
        self._standard_cols = []

        # use threshold
        for col in skew:
            if skew[col] >= self.threshold:
                self._log_cols.append(col)
            else:
                self._standard_cols.append(col)

        return self

    def transform(
        self,
        X: pd.DataFrame | np.ndarray | list,
        y: None = None,
    ) -> pd.DataFrame | np.ndarray:
        """Transform method"""

        _X = manage_input(X)
        _X = manage_columns(_X, self.fitted_columns)

        for col in self._log_cols:
            if not col in _X.columns:
                continue

            _X[col] = np.log1p(_X[col])

        return manage_output(_X, self.force_df_out)
```



### Describe alternatives you've considered, if relevant

- no ideas for now

### Additional context

A example notebook could be fond here :  [notebook](https://github.com/AlexandreGazagnes/scikit-transformers/blob/59677a96b737891bb3231da6fe79682e60b2e0db/docs/notebooks/transformer/LogColumnTransformer.ipynb)
user: AlexandreGazagnes
labels: Needs Triage: Issue requires triage
issue_28450: state: open
title: Add Examples to make_moons
body: Adding an example to the make_moons function, in response to #27982


#### What does this implement/fix? Explain your changes.
This PR adds an example to the make_moons function in sklearn.datasets._samples_generator.py. It checks off the make_moons tickmark in #27982
user: likeajumprope
issue_28448: state: open
title: DOC [PST] tune FAQ page styling
body: **Please note that this PR targets the `new_web_theme` branch!**

Towards #28084. This PR:

- Removes the `.. toctree::` in the page so that the question headings do not become (meaningless, in the sense that we have the secondary sidebar) links.
- Makes the questions (`h3` headings) rubric-like; this is personal taste so please let me know if maintainers think the original style is better.

| Current | This PR |
| ------- | ------- |
| ![image](https://github.com/scikit-learn/scikit-learn/assets/108576690/4a0d266e-8a18-4fae-b5f2-9921a6351148) | ![image](https://github.com/scikit-learn/scikit-learn/assets/108576690/30ef23eb-7fa6-4af0-b728-262db23d0bfe) |

user: Charlie-XIAO
issue_28443: state: open
title: ENH Plots for partial dependence and ICE of categoricals
body: ### Describe the workflow you want to enable

Having support of categorical features in `inspection.partial_dependence()` and `inspection.PartialDependenceDisplay()` is great. However, I would like to challenge the current visualization as **bar plot**.

- Bars take a lot of space.
- They suggest that 0 has a special meaning.
- They look dominant when a mix of categoricals and non-categoricals are plotted in the same plot.

Visualizating them as dots and/or lines would be neater in most cases. This would also fix the issue that "individual" ICE curves of categoricals are not implemented. There, we would simply use the same visualization (lines) as with non-categoricals.


### Describe your proposed solution

Replace bar plots by dots and/or lines. If individual curves are drawn, simply plot lines as for non-categoricals.

Centered ICE curves of multiple models would look like this:

![image](https://github.com/scikit-learn/scikit-learn/assets/16206095/68527977-6e8c-4a94-96df-9ec769ff5d67)


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: mayer79
labels: Needs Triage: Issue requires triage
issue_28442: state: open
title: [MRG] ENH: parallel RF oob scoring
body: * Fixes #28059

* use `joblib` to make oob predictions in random forest

* I also drafted in some `asv` benchmarks to try to capture the performance improvements more formally, but it looks like the `virtualenv` machinery for running `asv` may need a tune up? I kept seeing `Failed: trying different commit/environment` via `asv continuous -E virtualenv -e -b "RandomForestClassifierBenchmark" main treddy_issue_28059` and that was after having to apply a patch to `asv.conf.json` for a missing dependency (perhaps the team more commonly uses `conda` here)

* in lieu of the more formal benchmarks working (it would be good to confirm those...) I re-ran the code from the matching issue and the scaling does look better now with 16 cores on that machine:

<details>

![bench_feat](https://github.com/scikit-learn/scikit-learn/assets/7903078/c5a2fef3-9467-406a-8dec-fc19ac0d6c5b)

</details>
user: tylerjereddy
issue_28440: state: open
title: ENH allow to pass splitter for early stopping validation in HGBT
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Partially solves #18748. Alternative to #27124.

#### What does this implement/fix? Explain your changes.
This PR allows to pass splitters to parameter `validation_fraction` of `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor`.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: lorentzenchr
issue_28437: state: open
title: Allow boosting of estimators in scikit-learn pipelines using residuals
body: ### Describe the workflow you want to enable

I want to be able to use multiple estimators in one pipeline.  E.g.

```python
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import make_regression

X, y = make_regression(n_samples=100, n_features=10, noise=0.1)

pipeline = Pipeline([
    ('linear', LinearRegression()),
    ('gbm', GradientBoostingRegressor())
])

pipeline.fit(X, y)
```

This currently doesn't work: 
```python
TypeError: All intermediate steps should be transformers and implement fit and transform or be the string 'passthrough' 'LinearRegression()' (type <class 'sklearn.linear_model._base.LinearRegression'>) doesn't
```

This is for good reason: naively passing predict from the linear model to the fit for the gbm will lead to overfitting!

### Describe your proposed solution

Allow boosting estimators in scikit learn pipeline by subtracting the predict output from y at `fit` time and then adding the predict outputs at transform time. This enhancement would allow multiple regression estimators to be used consecutively, where each subsequent estimator fits on the residuals of the previous ones.

In my example above, at fit time the linear step proceeds as normal.  Then the gbm step detects 2 chained estimators, and subtracts the prediction from the linear step from y, and then fits to the residual.

At transform time, the predictions from the linear model would be added to the predictions from the gbm model.

To simplify things, I think you still wouldn't be allowed to connect an estimator to a transformer.

### Describe alternatives you've considered, if relevant

Vecstack is a good alternative, but its slow.  You have to cross-validate each estimator, which gets expensive if you chain more than 2 in a row.

```python
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import make_regression
from vecstack import StackingTransformer

X, y = make_regression(n_samples=100, n_features=10, noise=0.1)

pipeline = Pipeline([
    ('linear', StackingTransformer([('lr', LinearRegression())], regression=True, shuffle=True)),
    ('gbm', GradientBoostingRegressor())
])

pipeline.fit(X, y)
y_pred = pipeline.predict(X)
```

### Additional context

I love being able to use stacking inside sklearn pipelines.  It would be very cool to be able to use boosting too.  I haven't been able to figure out how to make a boosting class that adjusts y as it moves through the pipeline, which is why I think this feature would need to be implemented in the core library.
user: zachmayer
labels: Needs Triage: Issue requires triage
issue_28434: state: open
title: feat(Voting): Allow a `prefit` option to `VotingClassifier/Regressor`
body: #### Reference Issues/PRs
* Closes #12297

#### What does this implement/fix? Explain your changes.
This feature enables the user to pass in prefit estimators to both `VotingClassifier` and `VotingRegressor` by specifying `prefit=True`.

The `VotingX` is still required to have `fit()` called on it to do validation of the prefit estimators but it does not perform model fitting during this period.

Along with this, I've provided some basic first testing to ensure that the prefit estimators work equally if fit outside vs inside the Voter and validation checks during the call to `VotingX.fit()` to ensure that that prefit estimators are aligned in terms of the data that can flow through them.

#### Any other comments?
This is mainly a draft PR to backup [my comment](https://github.com/scikit-learn/scikit-learn/issues/12297#issuecomment-1947969744) left on issue #12297 with an initial implementation. There are likely a lot of missing features and issues that could arise.

Below is a list of issues that need to be resolved for this feature to integrate properly or issues that need to be discussed.

**Issues**
* When calling `VotingX(estimators, prefit=True).fit(X, y)`, should the `Voter` validate that the estimators are aligned with eachother, or with the passed in `X, y`, in terms of attributes like `n_features_in_` and `n_feature_names_`?
* What should be the defined `clone` behavior of this estimator, in relation [to this comment](https://github.com/scikit-learn/scikit-learn/issues/12297#issuecomment-427400873) by @amueller, linking to #8370
* What tags should be attached to the prefit version of a `Voter` with respect to the suite of estimator tests that exist for scikit-learn integrations.

As a meta-level question, should there exist a separate `PrefitVotingX` to alleviate some of this issues and simplify internals, at the expense of introducing a new estimator class?

---

### TODOs if accepted
* Add a `..versionadded:: ` to the docstrings
* Add a changelog entry
user: eddiebergman
issue_28428: state: open
title: DOC [PST] refactor API structure and improve display
body: **Please note that this PR targets the `new_web_theme` branch!**

Towards #28084.

- Detailed explanations of the changes: [#28428 (comment)](https://github.com/scikit-learn/scikit-learn/pull/28428#issuecomment-1948003147)
- Check the rendered docs [here](https://output.circle-artifacts.com/output/job/917e48f0-d243-4a59-bb41-8ec0f220fd3b/artifacts/0/doc/api/index.html)

Here is the list of things left to be done in this PR (if any). Maintainers please feel free to modify.

- [x] Placeholder.
user: Charlie-XIAO
issue_28422: state: open
title: MNT metadata routing: remove `MethodMapping.from_str()` and sort `caller`, `callee` in `MethodPair()`
body: #### What does this implement/fix? Explain your changes.

This PR aims to simplify some things about the development side of metadata routing: how routing and consuming methods are mapped together.

It does not change any functionality, but rather helps to make the code more readable.

1. `MethodMapping.from_str()` and it's usage are removed from the entire codebase. The alternative functionality, `MethodMapping.add()` that was existing next to it already before, is now used instead. This ensures consistency and clarity.

2. `caller` and `callee` in `MethodPair()` are consistently sorted so that `caller` comes first. These two have been kwarguments before, but putting caller always first supports code readability.

These should have been two separate PRs really, I recognise. Merging these two things together happened by accident.
user: StefanieSenger
issue_28421: state: open
title: FIX SplineTransformer does not output `nan` for constant features
body: #### Reference Issues/PRs
Fixes #26390

#### What does this implement/fix? Explain your changes.
Includes a check for constant features to prevent `nan` outputs.

#### Any other comments?
SplineTransformer does not give `nan` results with constant or near constant features:

```
import numpy as np
import sklearn
from sklearn.preprocessing import SplineTransformer

rng = np.random.RandomState(10)

a1 = SplineTransformer(extrapolation="periodic").fit_transform(np.ones(shape=(5, 1)))
a2 = SplineTransformer(extrapolation="periodic").fit_transform(np.ones(shape=(5, 1)) + rng.randn(5, 1) * 1e-16)
a3 =  SplineTransformer(extrapolation="periodic").fit_transform(np.ones(shape=(5, 1)) + rng.randn(5, 1) * 1e-17)

assert np.all(a1 == 0)
assert not np.all(a2 == 0)
assert np.all(a3 == 0)

print(a1)
print(a2)
print(a3)
```

Output:
```
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]]
[[0.33333333 0.66666667 0.         0.        ]
 [0.         0.         0.66666667 0.33333333]
 [0.33333333 0.66666667 0.         0.        ]
 [0.         0.         0.66666667 0.33333333]
 [0.         0.         0.66666667 0.33333333]]
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]]
```
user: rprkh
issue_28407: state: open
title: FIX Fix array API `train_test_split`
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Follow up to #26855


#### What does this implement/fix? Explain your changes.
This fixes the array API implementation of `train_test_split`. There were a few parts of `train_test_split` that appeared to work but didn't actually. 

#### Any other comments?

This includes all of #27904. Once it is merged this PR needs rebasing to remove those changes. The relevant changes are in the [final commit of this PR](https://github.com/scikit-learn/scikit-learn/pull/28407/commits/5d062ee7818a47cbd406e572fcb8c197d5dddc46).
<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: betatim
issue_28401: state: open
title: DOC [PST] fix dropdowns
body: **Please note that this PR targets the `new_web_theme` branch!**

Towards #28084. This mainly intends to fix the dropdowns. Note that there are many changes not closely related to the dropdown implementation itself, but are needed either to solve sphinx errors or keep consistency. The core changes are in `conf.py` and `sphinxext/dropdown_anchors.py`. [This page](https://output.circle-artifacts.com/output/job/ecb6c6dd-f947-48c8-a00f-10f11a9fcbbe/artifacts/0/doc/modules/linear_model.html) contains many dropdowns thus may be used as an example.

- More detailed explanations: [#28401 (comment)](https://github.com/scikit-learn/scikit-learn/pull/28401#issuecomment-1938035916)
- Check the rendered docs [here](https://output.circle-artifacts.com/output/job/c7cdc390-d0b6-4af3-b937-87d4aab9cabb/artifacts/0/doc/_changed.html), e.g., the [linear model page](https://output.circle-artifacts.com/output/job/c7cdc390-d0b6-4af3-b937-87d4aab9cabb/artifacts/0/doc/modules/linear_model.html)

Here is the list of things left to be done in this PR (if any). Maintainers please feel free to modify.

- [x] Placeholder.

@ArturoAmorQ who is in charge of #26617
@GaelVaroquaux who implemented the current `|details-start|` approach
@lucyleeow who opened #27127
@glemaitre who created `details-permalink.js`

I'm not sure if pinging maintainers like this would be annoying (please definitely let me know if it is). But if you have time and are interested please take a look. Thank you very much!
user: Charlie-XIAO
issue_28400: state: open
title: More intuitive check on GaussianMixture initialization
body: ### Describe the workflow you want to enable

Hey!

I am using GaussianMixture and testing different initialization methods (kmeans, random, k-means++ and random_from_data). When I ran the code using "init_params==kmeans++" an error was produced in sklearn/mixture/_base.py _initialize_parameters, saying that resposibilities were referenced before assignment. Of course this is because I initialized the GaussianMixture incorrectly and typed "kmeans++" instead of "k-means++" (without the dash).

### Describe your proposed solution

It would be great if init_params was checked upon initialization of GaussianMixture: something like "if self.init_params not in ['random', 'random_from_data', 'kmeans', 'k-means++']" would produce an error.

Alternatively (and maybe even more robust), one could add the extra else statement in _base.py _initialize_parameters to make sure the initialization technique is spelled correctly.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: rafalkowalewski1
labels: Needs Triage: Issue requires triage
issue_28399: state: open
title: FIX show only accuracy when having a subset of labels in classification report
body: #### Reference Issues/PRs

This PR fixes the issue #27927.

#### What does this implement/fix? Explain your changes.
There is an incosistency between the calculation of the micro-average in the code as compared to what the doc mentions. 
The doc mentions that:

> Micro average (averaging the total true positives, false negatives and false positives) is only shown for multi-label or multi-class with a subset of classes.

But the code even gives the micro-average for **_superset_** cases.

The fix is code is quite trivial and makes the code and documentation consistent. 

#### Any other comments?

The author of the issue shared some reproduction steps and the expected output. This fix is able to get to the expected output. 

`print(classification_report([0, 1], [1, 0], labels=[0, 1, 2], zero_division=0.0))`

Output:

![image](https://github.com/scikit-learn/scikit-learn/assets/110726731/a831af2c-6bb7-4748-9a4d-3a5a8fa2a01f)

user: vjoshi253
issue_28396: state: open
title:  Added verbose argument to permutation_importance API.
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
This change is for the request as mentioned in #28093.

#### What does this implement/fix? Explain your changes.
Added verbose argument to the permutation_importance API. The API now shows the progress messages for the parallel jobs when verbosity is given a non-zero value. 

![image](https://github.com/scikit-learn/scikit-learn/assets/110726731/587d7d1d-42fc-40bf-8313-e72092cfeedc)

#### Any other comments?

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: vjoshi253
issue_28394: state: open
title: Some ideas for breaking changes for 2.0
body: There's some breaking changes that we wanted to do for a long time but didn't do for 1.0 but I figure it might be worth starting a list somewhere for 2.0, since they keep coming up and we might forget them.

- in ``OneHotEncoder`` change ``handle_unknown`` to ``warn`` by default (and maybe recommend target encoder or max_categories or something)
- Make ``train_test_split`` assume the second entry is ``y`` and stratify if the estimator is a classifier, like all other ``cv`` methods do. Maybe call the attributes ``X`` and ``y`` and allow arbitrary additional entries.
- Make ``Pipeline`` do a ``clone`` in ``__init__``.

user: amueller
issue_28391: state: open
title: ⚠️ CI failed on Linux_Nightly_PyPy.pypy3 ⚠️
body: **CI is still failing on [Linux_Nightly_PyPy.pypy3](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=64564&view=logs&j=0b16f832-29d6-5b92-1c23-eb006f606a66)** (Feb 29, 2024)
Unable to find junit file. Please see link for details.
user: scikit-learn-bot
labels: Needs Triage: Issue requires triage
issue_28389: state: open
title: Apparent mismatch between possible arguments for `average` in the base stochastic gradient class
body: ### Describe the bug

Raised in https://github.com/scikit-learn/scikit-learn/pull/28373#discussion_r1482869890.

The  `average` parameter in [`BaseSGD`](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/linear_model/_stochastic_gradient.py#L82-L334) (which propagates to `SGDRegressor`, `SGDClassifier` and `SGDOneClassSVM`)  [is constrained](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/linear_model/_stochastic_gradient.py#L93) to non-negative integers or boolean values: `"average": [Interval(Integral, 0, None, closed="left"), bool, np.bool_]`.

This seems to be at odds with `average=0` seemingly meaning `average=True` which contradicts the typical truth-evaluation of `0`.


### Steps/Code to Reproduce

```
```

### Expected Results

```
```

### Actual Results

```
```

### Versions

This is in the main branch right now.


user: bmgdc
labels: Needs Triage: Issue requires triage
issue_28386: state: open
title: Proper sparse support in `IncrementalPCA`
body: ### Describe the workflow you want to enable

As of version 1.4 the implementation for PCA supports sparse inputs. That's grand, but the PCA implementation doesn't offer a `.partial_fit()` method. The `IncrementalPCA` does, but the way it handles sparse input is by casting it to dense first. This "works", but it may be more elegant/preformant to apply the same sparse trick to the `IncrementalPCA` implementation. 

### Describe your proposed solution

As the [release notes](https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_4_0.html#improved-memory-and-runtime-efficiency-for-pca-on-sparse-data) suggest it may just be a matter of investigating if the `scipy.sparse.linalg.LinearOperator` can be used here as well. But I'd need to dive in and check the implementation to understand if this is possible. 

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: koaning
labels: Needs Triage: Issue requires triage
issue_28375: state: open
title: ENH Add Friedman's H-squared
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs

Implements #22383

#### What does this implement/fix? Explain your changes.

@lorentzenchr 

This PR implements a clean version of Friedman's H^2 statistic of pairwise interaction strength. It uses a couple of tricks to speed up the calculations. Still, one needs to be cautious when adding more than 6-8 features. The basic strategy is to select e.g. the top 5 predictors via permutation importance and then crunch the corresponding pairwise (absolute and relative) interaction strength statistics.

(My) reference implementation: https://github.com/mayer79/hstats

#### Any other comments?

- The implementation also works for multi-output or multi-class classification.
- Plots might follow in a later PR.
- Univariate H-statistics also exist, but I have not added them (yet). They measure the proportion of prediction variability only explained by interactions involving feature j. We need to keep this in mind when thinking about the output API.

user: mayer79
issue_28368: state: open
title: Crash in T-SNE
body: ### Describe the bug

I got a crash using the (external) [hdbscan package](https://github.com/scikit-learn-contrib/hdbscan) in some special situation. I [debugged it](https://github.com/scikit-learn-contrib/hdbscan/issues/623) and found out that it happens in the scikit-learn package, specifically its T-SNE implementation. The hdbscan maintainer ([Leland McInnes](https://github.com/lmcinnes)!) suggested to report it here.

Is this something that should be guarded for in scikit-learn?

### Steps/Code to Reproduce

The simplest way to reproduce it (using the hdbscan package) is:
```py
import numpy as np
import hdbscan
model = hdbscan.HDBSCAN(gen_min_span_tree=True)
data = np.zeros((91, 3))
clustering = model.fit(data)
clustering.minimum_spanning_tree_.plot()
```
Note that it also happens when only a relative small proportion of points are equal (but only sometimes?), this is just the easiest way to show it. By default some warnings are displayed:
> ...\sklearn\decomposition\_pca.py:685: RuntimeWarning: invalid value encountered in divide
>   self.explained_variance_ratio_ = self.explained_variance_ / total_var
> ...\sklearn\manifold\_t_sne.py:1002: RuntimeWarning: invalid value encountered in divide
>   X_embedded = X_embedded / np.std(X_embedded[:, 0]) * 1e-4 

In the end it appears to be a problem in `sklearn.manifold._t_sne._barnes_hut_tsne.gradient()`, not (always?) being able to handle `nan` values. For example, this reproduces the crash:
```py
import numpy as np
from sklearn.manifold._t_sne import _barnes_hut_tsne
neighbors = np.array([1, 2, 0, 2, 0, 1], dtype='int64')
val_P = np.full_like(neighbors, 2 / 45, dtype='float32')
pos_output = np.full((3, 2), np.nan, dtype='float32')
forces = np.zeros_like(pos_output)
indptr = np.arange(7, step=2, dtype='int64')
_barnes_hut_tsne.gradient(val_P, pos_output, neighbors, indptr, forces, 0.5, 2, 11)
```
One layer deeper, the crash occurs inside `sklearn.neighbors._quad_tree._QuadTree.build_tree()`, as follows:
```py
import numpy as np
from sklearn.neighbors._quad_tree import _QuadTree
qt = _QuadTree(2, 11)
X = np.full((3, 2), np.nan, dtype='float32')
qt.build_tree(X)
```
The output of this (due to `verbose=11`) up to the crash is:
> [QuadTree] bounding box axis 0 : [nan, nan]
> [QuadTree] bounding box axis 1 : [nan, nan]
> [QuadTree] Inserting depth 0
> [QuadTree] inserted point 0 in cell 0
> [QuadTree] Inserting depth 0
> [QuadTree] inserted point 0 in new child 1
> [QuadTree] Inserting depth 0
> [QuadTree] Inserting depth 1
> ...
> [QuadTree] Inserting depth 6271
> [QuadTree] inserted point 0 in new child 6272
> [QuadTree] Inserting depth 6271
> [QuadTree] Inserting depth 6272

I didn't dig into the QuadTree code.

### Expected Results

I would expect anything but Python crashing.

### Actual Results

Python crashed: its console window just went away silently (on Windows).

### Versions

```shell
System:
    python: 3.12.1 (tags/v3.12.1:2305ca5, Dec  7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)]
executable: C:\Users\Public\Software\Python\python.exe
   machine: Windows-11-10.0.22631-SP0

Python dependencies:
      sklearn: 1.4.0
          pip: 23.3.2
   setuptools: 69.0.3
        numpy: 1.26.3
        scipy: 1.12.0
       Cython: 3.0.8
       pandas: 2.2.0
   matplotlib: 3.8.2
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: C:\Users\Public\Software\Python\Lib\site-packages\numpy.libs\libopenblas64__v0.3.23-293-gc2f4bdbb-gcc_10_3_0-2bde3a66a51006b2b53eb373ff767a3f.dll
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Cooperlake

       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: vcomp
       filepath: C:\Users\Public\Software\Python\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: C:\Users\Public\Software\Python\Lib\site-packages\scipy.libs\libopenblas_v0.3.20-571-g3dec11c6-gcc_10_3_0-c2315440d6b6cef5037bad648efc8c59.dll
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Cooperlake
```

user: mdruiter
issue_28367: state: open
title: unit test failures on x64 darwin and scipy 1.12
body: ### Describe the bug

after upgrading from scipy 1.11.4 -> 1.12 x64 darwin build fails with multiple unit test failures.
https://hydra.nixos.org/build/247540976
[raw log](https://cache.nixos.org/log/d449kn87qr1kcca138gd9pl14qg1nnxg-python3.11-scikit-learn-1.4.0.drv)

```console
=========================== short test summary info ============================
FAILED cluster/tests/test_mean_shift.py::test_parallel[float64] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED decomposition/tests/test_dict_learning.py::test_sparse_encode_shapes_omp - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED decomposition/tests/test_dict_learning.py::test_dict_learning_reconstruction_parallel - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED decomposition/tests/test_dict_learning.py::test_dict_learning_lassocd_readonly_data - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED decomposition/tests/test_dict_learning.py::test_sparse_coder_parallel_mmap - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED decomposition/tests/test_dict_learning.py::test_cd_work_on_joblib_memmapped_data - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED ensemble/tests/test_bagging.py::test_parallel_classification - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED ensemble/tests/test_bagging.py::test_parallel_regression - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED ensemble/tests/test_bagging.py::test_estimator - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED linear_model/tests/test_sgd.py::test_multi_core_gridsearch_and_early_stopping - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED ensemble/tests/test_forest.py::test_backend_respected - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED linear_model/tests/test_theil_sen.py::test_theil_sen_parallel - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED manifold/tests/test_mds.py::test_MDS - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED metrics/tests/test_classification.py::test_classification_metric_division_by_zero_nan_validaton[scoring0] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED metrics/tests/test_classification.py::test_classification_metric_division_by_zero_nan_validaton[scoring1] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED metrics/tests/test_classification.py::test_classification_metric_division_by_zero_nan_validaton[scoring2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED metrics/tests/test_classification.py::test_classification_metric_division_by_zero_nan_validaton[scoring3] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED ensemble/tests/test_voting.py::test_parallel_fit[42] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED metrics/tests/test_pairwise.py::test_sparse_manhattan_readonly_dataset[csr_matrix] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED metrics/tests/test_pairwise.py::test_sparse_manhattan_readonly_dataset[csr_array] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED feature_selection/tests/test_rfe.py::test_rfe_cv_n_jobs[42] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs[kernel0] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs[kernel1] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED model_selection/tests/test_search.py::test_random_search_bad_cv - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs[kernel2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs[kernel3] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED model_selection/tests/test_validation.py::test_cross_validate_many_jobs - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED model_selection/tests/test_validation.py::test_cross_validate_return_indices[42] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED neighbors/tests/test_kd_tree.py::test_kdtree_picklable_with_joblib[KDTree64] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED neighbors/tests/test_kd_tree.py::test_kdtree_picklable_with_joblib[KDTree32] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_permutation_importance_correlated_feature_regression[0.5-2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_permutation_importance_correlated_feature_regression[1.0-2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_robustness_to_high_cardinality_noisy_feature[0.5-2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_robustness_to_high_cardinality_noisy_feature[1.0-2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_permutation_importance_equivalence_sequential_parallel[500] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_permutation_importance_equivalence_sequential_parallel[1.0] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_permutation_importance_large_memmaped_data[array] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED neighbors/tests/test_neighbors.py::test_knn_forcing_backend[ball_tree-loky] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED neighbors/tests/test_neighbors.py::test_knn_forcing_backend[kd_tree-loky] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED neighbors/tests/test_neighbors.py::test_knn_forcing_backend[auto-loky] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_calibration.py::test_parallel_execution[True-sigmoid] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_calibration.py::test_parallel_execution[True-isotonic] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_calibration.py::test_parallel_execution[False-sigmoid] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_calibration.py::test_parallel_execution[False-isotonic] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED linear_model/tests/test_coordinate_descent.py::test_linear_models_cv_fit_with_loky[ElasticNetCV] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED linear_model/tests/test_coordinate_descent.py::test_linear_models_cv_fit_with_loky[LassoCV] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_config.py::test_config_threadsafe_joblib[loky] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_multioutput.py::test_multi_output_classification_partial_fit_parallelism - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_pipeline.py::test_feature_union_parallel - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED utils/tests/test_parallel.py::test_configuration_passes_through_to_joblib[loky-2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
```

### Steps/Code to Reproduce

run unit tests on x64 darwin with scipy 1.12

### Expected Results

unit tests pass

### Actual Results

https://cache.nixos.org/log/d449kn87qr1kcca138gd9pl14qg1nnxg-python3.11-scikit-learn-1.4.0.drv

### Versions

```shell
$ python -c 'import sklearn; sklearn.show_versions()'
System:
    python: 3.11.7 (main, Dec  4 2023, 18:10:11) [Clang 16.0.6 ]
executable: /nix/store/7l7g8ypn5aa6n3nvqids669rlzq9yi93-python3-3.11.7-env/bin/python3.11
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.4.0
          pip: None
   setuptools: None
        numpy: 1.26.2
        scipy: 1.12.0
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /nix/store/qws8gsrmyb3fks94qm4n63rgg97xa3v8-openmp-16.0.6/lib/libomp.dylib
        version: None
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /nix/store/mxlhcj24il7ncsmy89hv5h1vjgy5k1nr-openblas-0.3.25/lib/libopenblasp-r0.3.25.dylib
        version: 0.3.25
threading_layer: pthreads
   architecture: Prescott
    num_threads: 8
```

user: a-n-n-a-l-e-e
labels: Needs Reproducible Code: Issue requires reproducible code
issue_28352: state: open
title: BUG: use appropriate dtype in cv_results as opposed to always using object
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #28350. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
closes #28350


#### What does this implement/fix? Explain your changes.

Instead of always using dtype `object`, use a more appropriate dtype (the one detected by numpy)

#### Any other comments?

I noticed this when trying to use Polars, which is pickier about object dtype than pandas, for https://github.com/scikit-learn/scikit-learn/pull/28345

The existing tests already cover this functionality, so I've just updated them rather than increasing the test suite's running time. I can add a new test if desired though

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: MarcoGorelli
issue_28351: state: open
title: FEA Add d2_log_loss_score
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes: #20943

#### What does this implement/fix? Explain your changes.
- Adds d2_log_loss_score

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: OmarManzoor
issue_28350: state: open
title: GridSearchCV with PCA returns `object` masked array
body: ### Describe the bug

I noticed this while looking into https://github.com/scikit-learn/scikit-learn/pull/28345

The dtype of the `components_col` is `object`, which means that the pandas object which is then created is of dtype `object`.

### Steps/Code to Reproduce
```python
import numpy as np

from sklearn import datasets
from sklearn.decomposition import PCA
from sklearn.model_selection import GridSearchCV

pca = PCA()
X_digits, y_digits = datasets.load_digits(return_X_y=True)
param_grid = {"n_components": [5, 15]}
search = GridSearchCV(pca, param_grid)
search.fit(X_digits, y_digits)

print(search.cv_results_['param_n_components'].data.dtype)
```
### Expected Results

int64

### Actual Results

object

### Versions

```shell
System:
    python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
executable: /home/marcogorelli/tmp/.venv/bin/python3.10
   machine: Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.4.0
          pip: 22.0.2
   setuptools: 59.6.0
        numpy: 1.26.3
        scipy: 1.11.4
       Cython: None
       pandas: 2.2.0
   matplotlib: 3.8.2
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: /home/marcogorelli/tmp/.venv/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: libgomp
       filepath: /home/marcogorelli/tmp/.venv/lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: /home/marcogorelli/tmp/.venv/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: SkylakeX
```

user: MarcoGorelli
issue_28347: state: open
title: DOC [PST] version switcher and warning banner
body: **Please note that this PR targets the `new_web_theme` branch!**

Towards #28084. This main task of this PR is to configure the version switcher and the version warning banner.

- This PR mainly consists of two parts: [#28347 (comment)](https://github.com/scikit-learn/scikit-learn/pull/28347#issuecomment-1923722784)
- Check the rendered docs [here](https://output.circle-artifacts.com/output/job/12c0f63e-8ca9-4a16-931a-f945d7dccceb/artifacts/0/doc/index.html)

Here is the list of things left to be done in this PR (if any). Maintainers please feel free to modify.

- [x] Placeholder.
user: Charlie-XIAO
labels: Waiting for Second Reviewer: First reviewer is done, need a second one!
issue_28344: state: open
title: added example for sklearn.datasets.fetch_rcv1
body: #27982 
user: Higgs32584
issue_28343: state: open
title: added example for sklearn.datasets.fetch_20newsgroups
body: #27982 
user: Higgs32584
issue_28342: state: open
title: Added example for sklearn.datasets.make_biclusters
body: #27982
user: Higgs32584
issue_28341: state: open
title: RFC switch to Polars as the default dataframe lib in our examples
body: We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.
user: adrinjalali
issue_28339: state: open
title: Special characters (e.g. &) are not escaped by sklearn.tree.export_graphviz
body: ### Describe the bug

Exporting a decision tree where the `feature_names` or `class_names` contain special characters (particularly `&<>`) results in invalid graphviz output, as those characters have specific meanings to graphviz. Escaping to `&amp;`, `&lt;` and `&gt;` results in correct output. This can of course be done by the user but it's something I think scikit-learn should handle internally.

### Steps/Code to Reproduce

```python
from sklearn.datasets import load_iris
from sklearn import tree
iris = load_iris()
clf = tree.DecisionTreeClassifier()
clf = clf.fit(iris.data, iris.target)

target_names = ["setosa & 123", "versicolor", "virginca"]
# target_names = ["setosa &amp; 123", "versicolor", "virginca"]  # This one works

tree.export_graphviz(
		clf,
		out_file="tree.dot",
		feature_names=iris.feature_names,
		class_names=target_names,
		filled=True,
		special_characters=True,
		)

```

Then run graphviz

```bash
dot tree.dot -Tsvg -o tree.svg 
```

### Expected Results

Graphviz successfully converts to SVG without error.

### Actual Results

```
Error: not well-formed (invalid token) in line 1 
... <br/>class = setosa & 123 ...
in label of node 0
Error: not well-formed (invalid token) in line 1 
... <br/>class = setosa & 123 ...
in label of node 1
```

Although SVG output is written to disk it is not correct.
![image](https://github.com/scikit-learn/scikit-learn/assets/8050853/8aa517f5-9764-4d9a-93f9-d20864f6085c)


### Versions

```shell
System:
    python: 3.8.10 (default, Nov 22 2023, 10:22:35)  [GCC 9.4.0]
executable: /home/domdf/Python/01 GitHub Repos/13 GunShotMatch/gunshotmatch-cli/venv/bin/python3
   machine: Linux-5.15.0-92-generic-x86_64-with-glibc2.29

Python dependencies:
      sklearn: 1.3.2
          pip: 23.3.2
   setuptools: 69.0.3
        numpy: 1.24.4
        scipy: 1.10.1
       Cython: None
       pandas: 2.0.3
   matplotlib: 3.7.4
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: libgomp
       filepath: /home/domdf/Python/01 GitHub Repos/13 GunShotMatch/gunshotmatch-cli/venv/lib/python3.8/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: /home/domdf/Python/01 GitHub Repos/13 GunShotMatch/gunshotmatch-cli/venv/lib/python3.8/site-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so
        version: 0.3.21
threading_layer: pthreads
   architecture: Zen

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: /home/domdf/Python/01 GitHub Repos/13 GunShotMatch/gunshotmatch-cli/venv/lib/python3.8/site-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Zen
```

user: domdfcoding
issue_28337: state: open
title: Enforce `feature_names_in_` and `n_features_in_` in `check_estimator` post SLEP007 implementation
body: ### Describe the workflow you want to enable

I would like to propose an enhancement to the `check_estimator` function, particularly in light of the implementation of SLEP007. As per SLEP007, which was integrated into scikit-learn starting from version 1.1.0, estimators are expected to support `feature_names_in_` and `n_features_in_` attributes. However, it appears that the `check_estimator` utility does not currently enforce the presence of these attributes.

### Describe your proposed solution

I propose that the `check_estimator` function be updated to include checks for the existence of the `feature_names_in_` and `n_features_in_` attributes.

### Additional context

- SLEP007:[ https://scikit-learn-enhancement-proposals.readthedocs.io/en/latest/slep007/proposal.html](https://scikit-learn-enhancement-proposals.readthedocs.io/en/latest/slep007/proposal.html)
- Version of scikit-learn where SLEP007 was implemented: 1.1.0
- Example of `check_estimator` not enforcing these attributes: https://github.com/microsoft/LightGBM/issues/6279
user: nicklamiller
issue_28322: state: open
title: Fixed TimeSeriesSplit training set size formula
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes #28321.


#### What does this implement/fix? Explain your changes.
This fixes the incorrect behavior of the formula to compute the training set size for `TimeSeriesSplit`

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: ZiadAmerr
issue_28321: state: open
title: TimeSeriesSplit Train Size formula correction
body: ### Describe the issue linked to the documentation

The documentation states:
> The training set has size `i * n_samples // (n_splits + 1) + n_samples % (n_splits + 1)` in the `i`th split, with a test set of size `n_samples // (n_splits + 1)` by default, where `n_samples` is the number of samples.

The equation for the training set looks like it is flawed. Given a split with:

```python
test_set_size = 10
n_splits = X.shape[0] // test_set_size
max_train_size = 10000

cv = TimeSeriesSplit(n_splits=n_splits, test_size=test_size, max_train_size=max_train_size)

pairs
for i, (train_index, test_index) in enumerate(cv.split(X)):
    pairs.append([i, len(train_index), len(test_index)])
```

The previous code generates a list similar to this:
```
[[0, 7, 10],
[1, 17, 10],
[2, 27, 10],
[3, 37, 10],
...
[996, 9967, 10],
[997, 9977, 10],
[998, 9987, 10],
[999, 9997, 10],
[1000, 10000, 10],
[1001, 10000, 10],
...
[3177, 10000, 10],
[3178, 10000, 10]]
```

I tried using the formula in the documentation to determine the size of the training set at a given iteration:
```python
def get_train_size(i, n_samples, n_splits):
    return i * n_samples // (n_splits + 1) + n_samples % (n_splits + 1)

get_train_size(0, 31797, 3179)  # 3177, which is incorrect, it should be 7!


def get_train_size(i, n_samples, n_splits, max_train_size=float("inf")):
    start_value = n_samples % n_splits

    increment = n_samples // n_splits

    return int(min(start_value + i * increment, max_train_size))

get_train_size(0, 31797, 3179)  # 7, correct
get_train_size(999, 31797, 3179)  # 9997, correct
get_train_size(1000, 31797, 3179)  # 10007, correct
get_train_size(1000, 31797, 3179, 10000)  # 10000, correct
```

### Suggest a potential alternative/fix

So the correction to the formula should be that the training set size is equal to:
`i * (n_samples // n_splits) + n_samples % n_splits`
instead of:
`i * n_samples // (n_splits + 1) + n_samples % (n_splits + 1)`
user: ZiadAmerr
labels: Needs Triage: Issue requires triage
issue_28320: state: open
title: FIX an issue with Halving*SearchCV where n_possible_iterations is short by 1
body: Fixes #27422. 
user: adrinjalali
issue_28314: state: open
title: Request to update "Choosing the Right Estimator" Graphic (scikit-learn algorithm cheat sheet)
body: ### Describe the issue linked to the documentation

As seen here:
https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html

One of the "tough luck" paths that go through the clustering section appear to say this is the case when there are >10k samples. 

### Suggest a potential alternative/fix

However, with modern computational hardware, and the optimized implementation of DBSCAN in Scikit-learn, it appears that it may be helpful to recommend DBSCAN as a possible solution for datasets containing <100K or even <1M datapoints for clustering in reasonable amounts of time on CPU.
user: joshglens
issue_28300: state: open
title: #27982 added examples for sklearn.datasets.fetch_20newsgroups_vectorized sklearn.datasets.fetch_kddcup99 sklearn.datasets.fetch_lfw_pairs sklearn.datasets.fetch_lfw_people sklearn.datasets.fetch_olivetti_faces sklearn.datasets.load_linnerud  sklearn.datasets.load_svmlight_files
body: #27982
added examples for

sklearn.datasets.fetch_20newsgroups_vectorized
sklearn.datasets.fetch_kddcup99
sklearn.datasets.fetch_lfw_pairs
sklearn.datasets.fetch_lfw_people
sklearn.datasets.fetch_olivetti_faces
sklearn.datasets.load_linnerud 
sklearn.datasets.load_svmlight_files

NOTE: Some of the examples, specifically 
fetch_lfw_people
fetch_lfw_pairs
were copied over from the page 

https://scikit-learn.org/stable/datasets/real_world.html#newsgroups-dataset


This was supposed to be two branches but I forgot to make another one my bad :)



user: Higgs32584
issue_28288: state: open
title: #27982 Created examples for sklearn.linear_model.enet_path sklearn.linear_model.lars_path sklearn.linear_model.lars_path_gram sklearn.linear_model.orthogonal_mp sklearn.linear_model.orthogonal_mp_gram
body: #27982 
Created examples for 
sklearn.linear_model.enet_path
sklearn.linear_model.lars_path
sklearn.linear_model.lars_path_gram
sklearn.linear_model.orthogonal_mp
sklearn.linear_model.orthogonal_mp_gram



user: Higgs32584
issue_28286: state: open
title: #27982 added examples for sklearn.utils.sparsefuncs_fast.inplace_csr_row_normalize_l1 and sklearn.utils.sparsefuncs_fast.inplace_csr_row_normalize_l2
body: #27982
user: Higgs32584
issue_28285: state: open
title: DOC and DOCSTRING Examples for sklearn.feature_extraction.image.grid_to_graph and sklearn.feature_extraction.image.img_to_graph
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
This PR fixes part of #27982

#### What does this implement/fix? Explain your changes.
Have added examples for sklearn.feature_extraction.image.grid_to_graph and sklearn.feature_extraction.image.img_to_graph in the docstrings. Also added some explanation about connectivity matrix for sklearn.feature_extraction.image.grid_to_graph.
![image](https://github.com/scikit-learn/scikit-learn/assets/67852834/dd34d0fd-b479-4f45-b4fd-66b85ead9594)

please let me know if any further changes needed as I am new here. Thanks much!
@glemaitre 

#### Any other comments?


user: Dion11235
issue_28280: state: open
title: Tests failing when cuda installed but no GPU is present
body: after doing `conda install pytorch cupy`, my tests fail with:

```
FAILED sklearn/metrics/tests/test_common.py::test_array_api_compliance[
accuracy_score-check_array_api_binary_classification_metric-cupy-None-None] 
- cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorNoDevice: 
- no CUDA-capable device is detected
```

I don't think tests should ever fail for this, should they?

cc @ogrisel @betatim 
user: adrinjalali
issue_28274: state: open
title: Trigger lockfile update with a comment
body: I've never been able to run the script to update the lock files w/o errors. My last attempt resulted in https://github.com/scikit-learn/scikit-learn/pull/28258#issuecomment-1910627538 which also didn't work, and that's after I had to install conda on my env, which I don't usually have since I use micro mamba.

It would be nice to be able to trigger a bot to update lock files the same way you can re-render on conda-forge. A comment with value similar to `@scikit-learn-bot please re-render` could update the lock files on the PR.

WDYT @lesteve ?
user: adrinjalali
issue_28268: state: open
title: [DRAFT] Support missing-values in ExtraTrees
body: #### Reference Issues/PRs
Follow-up to: #27966 

Closes: #27931 


#### What does this implement/fix? Explain your changes.
- Adds support and tests for missing-values in the `ExtraTrees*`

#### Any other comments?

Waiting for #27966 to be merged first. This just shows the relevant code changes that introduces the missing-value support overall for `ExtraTrees`. Note the bulk of the work for going from tree->forest missing-value support was added in https://github.com/scikit-learn/scikit-learn/pull/26391.  The main change here is to introduce certain unit tests.

user: adam2392
issue_28244: state: open
title: FEAT Introduce DBCV as new cluster metric
body: for theoretical details/high level considerations see #28243


The version of DBCV which I want to integrate is the one shipped with the validity module (which I contributed to in the past) of [this](https://github.com/scikit-learn-contrib/hdbscan) version of HDBSCAN. (the HDBSCAN implementation of that repo was [recently integrated into scikit-learn](https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_3_0.html#hdbscan-hierarchical-density-based-clustering))


steps before merging:
- integrate code from HDBSCAN into the existing module structure:
  - [x] add code from `github.com/scikit-learn-contrib/hdbscan/blob/master/hdbscan/validity.py`
  - [x] sort out imports
  - [x] renaming
- adopt validation constructs specific/internal to sklearn:
  - [x] utilize `validate_params` decorator
  - [x] run input through `check_X_y`
  - [x] `check_number_of_labels`
- make the new code conform to usual cluster metric "API" input assumptions/interface
  - [x] make validation calculations agnostic of label input type via usage of `LabelEncoder`
  - [x] allow `list` input for `labels` param instead of assuming `numpy.ndarray`
- add tests:
  - [x] register new public DBCV function for param validation checking at `/tests/test_public_functions.py`
  - [x] register new logic for generic, unsupervised cluster metric testing at `/metrics/cluster/tests/test_common.py`
  - [x] write DBCV-specific test checking score output at `/metrics/cluster/tests/test_unsupervised.py` 
- documentation:
  - [x] information about params and output in docstring (already given per HDBSCAN implementation)
  - [x] add a release note


Resolves #28243
user: luis261
issue_28225: state: open
title: Added a way to choose how sample_y in GP regression samples the multivariate normal
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### What does this implement/fix? Explain your changes.

In the GP regression `sample_y` method, `rng.multivariate_normal` is called with its default `method` argument, i.e. "svd". This is fine when `sample_y` is called on a small number of points, but I found that for a large set of evaluation points, this could get quite costly. 

Using the example in the docs,

```python 
X = np.linspace(start=0, stop=10, num=1_000).reshape(-1, 1)
y = np.squeeze(X * np.sin(X))

gpr = GaussianProcessRegressor(Matern(nu=2.5))

_ = gpr.fit(X,y)

x_sample = np.random.uniform(0,10,1000)
_ = gpr.sample_y(x_sample)
# takes 0.6 seconds, of which 0.4 is in multivariate_normal 

x_sample = np.random.uniform(0,10,5000)
_ = gpr.sample_y(x_sample)
# takes 38 seconds, of which 36 is in multivariate_normal 
```
Repeating the same exercise with `method = "cholesky"` in `rng.multivariate_normal` reduces the runtime to a much more reasonable 2.5 seconds. 


#### Any other comments?

I kept the default to "svd" so users shouldnt see any change unless they opt in. 

user: varchasgopalaswamy
issue_28219: state: open
title: Euclidean Specialization for ArgKminClassMode
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Towards #25888

#### What does this implement/fix? Explain your changes.
- Implements the Euclidean Specialization for ArgKminClassMode

#### Any other comments?
CC: @jjerphan @Micky774 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: OmarManzoor
issue_28208: state: open
title: Use modern `setuptools`' PEP 517 build backend explicitly
body: Without an explicit entry, the `__legacy__` fallback is used, that behaves slightly differently. In the long run, it's best to always have it set.
user: webknjaz
issue_28205: state: open
title: ENH Add metadata routing for `FeatureUnion`
body: #### Reference Issues/PRs
Towards #22893

#### What does this implement/fix? Explain your changes.
This PR adds metadata routing to `FeatureUnion` and the corresponding tests.

I was confused about why we cannot do `set_fit_transform_request`. I know we can work around it, but from a user's perspective it's not self-explanatory.
user: StefanieSenger
issue_28191: state: open
title: ENH: use the sparse-sparse backend for computing pairwise distance
body: First reported in: https://github.com/scikit-learn-contrib/imbalanced-learn/issues/1056

We have a regression in `kneighbors` with sparse matrix from 1.1.X to 1.3.X.
A code sample to reproduce:

```python
# %%
import sklearn
sklearn.__version__

# %%
import numpy as np
from scipy import sparse
from sklearn.neighbors import KNeighborsRegressor

n_samples, n_features = 1_000, 10_000
X = sparse.random(n_samples, n_features, density=0.01, format="csr", random_state=0)
rng = np.random.default_rng(0)
y = rng.integers(0, 2, size=n_samples)
knn = KNeighborsRegressor(n_neighbors=5).fit(X, y)

# %%
%%timeit
knn.kneighbors(X, return_distance=False)
```

#### 1.1.X

```
21.5 ms ± 217 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

#### `main`

```
1.16 s ± 9.87 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

#### Small benchmark

![image](https://github.com/scikit-learn/scikit-learn/assets/7454015/5abe644f-5c05-4124-ab43-dd0a1cdd3e58)

<details>

```python
# %%
import sklearn
sklearn.__version__

# %%
import time
from collections import defaultdict
import numpy as np
from scipy import sparse
from sklearn.neighbors import KNeighborsRegressor

n_samples, n_features = 1_000, [500, 1_000, 2_500, 5_000, 7_500, 10_000, 25_000, 50_000]

results = defaultdict(list)
for nf in n_features:
    X = sparse.random(n_samples, nf, density=0.01, format="csr", random_state=0)
    rng = np.random.default_rng(0)
    y = rng.integers(0, 2, size=n_samples)
    knn = KNeighborsRegressor(n_neighbors=5).fit(X, y)
    start = time.time()
    knn.kneighbors(X, return_distance=False)
    elapsed_time = time.time() - start
    results["version"].append(sklearn.__version__)
    results["n_features"].append(nf)
    results["elapsed_time"].append(elapsed_time)

# %%
import pandas as pd

results = pd.DataFrame(results)
results.to_csv(f"bench_{sklearn.__version__}.csv", index=False)
```

```python
# %%
import pandas as pd

results_main = pd.read_csv(f"bench_1.5.dev0.csv")
results_1_1 = pd.read_csv(f"bench_1.1.3.csv")
results = pd.concat([results_main, results_1_1], axis=0)
results

# %%
import seaborn as sns

sns.set_context("talk")

# %%
sns.relplot(
    data=results,
    x="n_features",
    y="elapsed_time",
    hue="version",
    kind="line",
    height=6,
    aspect=1.5,
    legend=True,
)
sns.despine(offset=10, trim=True)
```

</details>

#### Debug profiling

My first investigation look like we are spending all our time in the following function:

https://github.com/scikit-learn/scikit-learn/blob/5c7e831306e0a087c2b6af6913fa5b3c402f6d67/sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.pyx.tp#L39-L69

But I'm a bit rusty with profiling native code. I need a bit more time to investigate.
user: glemaitre
issue_28189: state: open
title: DOC update model persistence page to provide clearer recommendations
body: #### Reference Issues/PRs
Fixes #27514

#### What does this implement/fix? Explain your changes.
Updates the model persistence page and provides more information on the different approaches used to persist models.

#### Any other comments?
Based on https://github.com/scikit-learn/scikit-learn/issues/27514#issue-1920970883, the tasks to be completed as part of this PR are as follows:

- [x] List all alternatives near the beginning of the document.
- [x] Start by discussing the fact that you need to pin all transitive scikit-learn dependencies to be able to safely load a pickle, which is the format most people will use, and probably gets you 80% of the benefit you will get from more complicated recommendations.
- [x] Do more to compare the presented alternatives (what does skops lack that would make one want to use ONNX?)
- [x] Summarize the most important points at the end of the document.
user: rprkh
issue_28179: state: open
title: ENH verbose >= 2 for per iteration info in HGBT
body: #### Reference Issues/PRs
None

#### What does this implement/fix? Explain your changes.
This PR gives finer control over verbosity in `HistGradientBoostingClassifier` and
`HistGradientBoostingRegressor`, i.e. the per iteration info is only emitted for `verbose >= 2`.

#### Any other comments?

user: lorentzenchr
issue_28178: state: open
title: Exception in LogisticRegressionCV
body: ### Describe the bug

The code provided below raises ValueError. I guess that the problem is that minor classes may not be included in **train** or **val** sets for some folds during internal cross-validation, even with stratified split. This produces errors with some metrics other than default (accuracy).

One solution may be setting log-proba to -inf for classes not present in the train set, as well as providing label argument. How can I fix this in the most simple way?

### Steps/Code to Reproduce

```
import numpy as np
from sklearn.linear_model import LogisticRegressionCV
X = np.zeros((10, 1))
y = [1, 1, 1, 1, 1, 2, 2, 2, 2, 3]
logreg = LogisticRegressionCV(cv=5, scoring='neg_log_loss')
logreg.fit(X, y)
```

### Expected Results

No exception thrown

### Actual Results

ValueError: y_true and y_pred contain different number of classes 2, 3. Please provide the true labels explicitly through the labels argument. Classes found in y_true: [0 1]

### Versions

```shell
System:
    python: 3.10.8 (main, Nov  2 2023, 15:57:09) [GCC 9.4.0]
executable: /data/osedukhin/tabular-models/venv/bin/python
   machine: Linux-5.4.0-123-generic-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.4.0
          pip: 22.2.2
   setuptools: 63.2.0
        numpy: 1.26.2
        scipy: 1.11.3
       Cython: None
       pandas: 2.1.3
   matplotlib: 3.8.1
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: /data/osedukhin/tabular-models/venv/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: openmp
   internal_api: openmp
    num_threads: 32
         prefix: libgomp
       filepath: /data/osedukhin/tabular-models/venv/lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: /data/osedukhin/tabular-models/venv/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: SkylakeX
```

user: sedol1339
issue_28175: state: open
title: Inconsistency in `DecisionTreeClassifier` Threshold Behavior
body: ### Describe the bug

I've encountered an unexpected behavior in `DecisionTreeClassifier` when using a decision stump (a tree with one root node and two leaf children). My assumption is based on the standard decision tree logic where a feature value `x` is classified to the left child if `x <= threshold`. Therefore, I expect the following assertion to always be true:
```python
assert clf.apply([[clf.tree_.threshold[0]]]) == 1
```

This assertion is meant to test that a feature value equal to the root node's threshold is classified to the left child node, in accordance with the `x <= threshold` rule. However, I have observed that in approximately 25% of the cases, the data point is unexpectedly classified to the right child node instead of the left.

The issue persists regardless of whether the `threshold[0]` is explicitly converted to `float32` or not. Given scikit-learn's documentation stating that
> All decision trees use `np.float32` arrays internally. If training data is not in this format, a copy of the dataset will be made

this behavior is puzzling. Precise thresholding is crucial for my application. Thank you for your time and effort in maintaining this important library and for any insights you can provide regarding this issue.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.tree import DecisionTreeClassifier

correct, wrong = [], []
n_trials = 500
for seed in range(n_trials):
    rand = np.random.RandomState(seed)
    X = rand.normal(size=(100, 1))
    y = rand.randint(0, 2, size=X.shape[0])
    clf = DecisionTreeClassifier(max_depth=1, random_state=0).fit(X, y)
    
    thres = clf.tree_.threshold[0]
    # thres = np.float32(thres) # gives the same correct-wrong distribution
    if clf.apply([[thres]]) == 1:
        correct.append(thres)
    else:
        wrong.append(thres)

    assert clf.apply([[np.nextafter(np.float32(thres), np.float32(-np.inf))]]) == 1 # this is true though
print(f'{len(correct) / n_trials:%}') # 75%
print(f'{len(wrong) / n_trials:%}') # 25%
```

### Expected Results

```
100.0000%
0.0000%
```

### Actual Results

```
75.0000%
25.0000%
```

### Versions

```shell
System:
    python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
executable: /usr/bin/python3
   machine: Linux-6.1.58+-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.2
   setuptools: 67.7.2
        numpy: 1.23.5
        scipy: 1.11.4
       Cython: 3.0.8
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/numpy.libs/libopenblas64_p-r0-742d56dc.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 2
         prefix: libgomp
       filepath: /usr/local/lib/python3.10/dist-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Haswell
```

user: fanurs
issue_28169: state: open
title: RFC Expose a plublic method to compute the objective function
body: I think that it would be valuable that all estimators that optimize some objective function expose a public method to compute it.
My main motivation is for the callbacks, for early stopping or monitoring, but I'm sure it would be useful in other contexts.

To really be practical, the signature should be the same across all estimators. What I have in mind is something like:

```py
def objective_function(self, X, y=None, *, sample_weight=None, normalize=False):
    y_pred = self.predict(X, y)
    # or Xt = self.transform(X) because some transformers do optimize an objective function.

    data_fit = <computation of the data fit term>
    penalization = <computation of the penalization term>

    if normalize:                     # allow to return a per sample
        data_fit /= X.shape[0]        # objective function
        penalization /= X.shape[0]    # for convenience
        # X.shape[0] probably needs to be replaced by sample_weight.sum()
 
    return data_fit + penalization, data_fit, penalization
```

If we want to compute the objective function of the training set during fitting, we could allow to provide the current variables that are required to compute it at a given iteration, encapsulated in a single argument (a dict):
```py
def objective_function(self, X, y=None, *, sample_weight=None, fit_state=None, normalize=False):
    if fit_state is None:
        y_pred = self.predict(X)
    else:
        y_pred = <compute y_pred using the information in fit_state>

    ...
```
where the content of fit_state would be estimator specific, and detailed in the docstring of `objective_function` for each estimator.
user: jeremiedbb
labels: Hard: Hard level of difficulty
Meta-issue: General issue associated to an identified list of tasks
issue_28151: state: open
title: RFC: Towards reproducible builds for our PyPI release wheels
body: Given the popularity of our project, it is possible that our release automation is considered an interesting target to conduct supply chain attacks to make our binaries ship spyware or ransomware to some of our users.

One way to detect such attacks would be to:

- make sure we produce [reproducible builds](https://reproducible-builds.org/);
- rebuild our wheels from independent build environments and check that we obtain the same hash as the for binaries obtained by our release CI to make sure that our release CI environment has not be tempered to inject malware in our binaries;
- optionally make it possible to publish GPG signed statements that some released artifact digests were successfully byte-for-byte reproduced from source independently.

The first step would to make our wheels as reproducible as possible would be to define deterministic values for the `SOURCE_DATE_EPOCH` (and maybe `PYTHONHASHSEED`, that cannot hurt) environment variables.

However this would not be enough.

To get this fully work as expected, we would also need to guarantee that:

 - we use recent enough versions of pip/setuptools/wheel/auditwheel/delocate
   that honor `SOURCE_DATE_EPOCH`;

 - a full description of the build environment (e.g. versions and sha256 digests of the
   compilers and other build dependencies) is archived in our source repo for a given tag
   of scikit-learn. Ideally, all those build dependencies should themselves be
   byte-for-byte reproducible from their own public source code repo.

At the moment, some build dependencies such as NumPy and Cython come from the `pyproject.toml` file which only specifies a minimum version. This means that we may end up with a newer versions of these dependencies than the one used to build the wheels for a given tag. `cibuildwheel` itself is not pinned, hence neither the dependencies it installs in its managed venvs (pip, setuptools, wheel, auditwheel, delocate).

Furthermore, we do not archive or pin the versions and sha256 digests of the compilers yet. For Linux, this depends on the manylinux docker image used by cibuildwheel, which at the time of writing, is not guaranteed to be reproducible, even when using the same docker image tag. For windows and macos, the compilers come from the VM image used on our CI which we do not archive neither their version numbers nor the hash of their binaries.

Ideally all this information should be in our source code at the time of the release (reachable via a checkout of our commit tag).

Finally, we might need to set a specific `umask`:

- https://github.com/pypa/wheel/issues/362

Not sure about how to get deterministic file permission metadata for macOS and Windows wheels.

Finally, once our builds are made 100% reproducible, we would need to document:

- document instructions (and provide official scripts) to allow anyone to easily re-build the binaries independently from source;
- publish official reproducibility results on a public site, ideally not only on our github pages hosted website and maybe sign those with GPG for instance;
- coordinate with people from https://scientific-python.org or even the PSF or pypi.org admins to define and follow community-wide best practices.

Doing all of this will require significant maintainers time investment, but we can probably start from low hanging fruits such as setting `SOURCE_DATE_EPOCH` in our release CI scripts.
user: ogrisel
labels: Meta-issue: General issue associated to an identified list of tasks
issue_28139: state: open
title: LedoitWolf.fit() crashes Jupyter Notebook 
body: ### Describe the bug

Hi, when i used LedoitWolf.fit(), my Jupyter Notebook crashes immediately. I am using the toy example given in the source code of 
LedoitWolf class


### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.covariance import LedoitWolf
real_cov = np.array([[.4, .2],[.2, .8]])
np.random.seed(0)
X = np.random.multivariate_normal(mean=[0, 0],cov=real_cov,size=50)
cov = LedoitWolf().fit(X)
```

### Expected Results

It is exptected to fit the estimator.

### Actual Results

However kernel crashes. sklearn.show_versions() also produce a WinError.

### Versions

```shell
Sklearn version is 1.3.2
```

user: gururgg
issue_28130: state: open
title: Expanded ColumnTransformer functionality -- transforming subsets of data
body: ### Describe the workflow you want to enable (edited)

the ability to (inverse_)transform data corresponding to a subset of the ColumnTransformer's component transformations

### Describe your proposed solution

Data of a smaller size can be passed in with a new keyword that identifies the relevant component transformations by name

### Describe alternatives you've considered, if relevant

a function that subsets a ColumnTransformer object, including adjusting the column numbers

### Additional context

In artificial intelligence applications, the researcher may want to transform an entire dataset with column groups for learning, but then transform new data corresponding just to interventions or predictions using the same transformations at a later time. Hence, the need to be able to subset a ColumnTransform or pass in only a part of the data.

See #27957 for more discussion.
user: rebeccaherman1
issue_28108: state: open
title: BUG Wrong error raised for OneVsRestClassifier with a sub-estimator that doesn't allow partial_fit
body: ### Describe the bug

When using the `OneVsRestClassifier` with a sub-estimator, that doesn't have `partial_fit` implemented, a misleading error message is shown: `AttributeError: This 'OneVsRestClassifier' has no attribute 'partial_fit'`.

Though, `OneVsRestClassifier` does implement `partial_fit`, but the underlying estimator doesn't.

There is an appropriate error raising already implemented in
https://github.com/scikit-learn/scikit-learn/blob/f1e89363f6777155a25b3574db9f0fc5c21a8c51/sklearn/multiclass.py#L437

```python
            if not hasattr(self.estimator, "partial_fit"):
                raise ValueError(
                    ("Base estimator {0}, doesn't have partial_fit method").format(
                        self.estimator
                    )
                )
```
But it's not run, because the AttributeError from the `@available_if` decorator/descriptor thing pops up earlier.

I'd like to learn about that, repair that and add a test to make sure this doesn't happen again by accident.
I will also check if other methods from the multiclass classifiers are also affected.

Is it alright if I go ahead with this?

### Steps/Code to Reproduce

```python
from sklearn.datasets import load_iris
from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import LogisticRegression
import numpy as np

iris = load_iris()
sample_weight = np.ones_like(iris.target, dtype=np.float64)
clf = OneVsRestClassifier(
estimator=LogisticRegression(random_state=42)
)
clf.partial_fit(iris.data, iris.target)
```

### Expected Results

`ValueError:  LogisticRegression doesn't have partial_fit method`

### Actual Results

`AttributeError: This 'OneVsRestClassifier' has no attribute 'partial_fit'`.

### Versions

```shell
System:
    python: 3.10.6 (main, Oct 10 2022, 12:43:33) [GCC 9.4.0]
executable: /home/stefanie/.pyenv/versions/3.10.6/envs/scikit-learn_dev/bin/python
   machine: Linux-5.15.0-91-generic-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.dev0
          pip: 23.3.2
   setuptools: 63.2.0
        numpy: 1.26.0
        scipy: 1.11.3
       Cython: 0.29.34
       pandas: 2.1.4
   matplotlib: 3.8.0
       joblib: 1.3.2
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/stefanie/.pyenv/versions/3.10.6/envs/scikit-learn_dev/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/stefanie/.pyenv/versions/3.10.6/envs/scikit-learn_dev/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 8

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
        version: None
    num_threads: 8
```

user: StefanieSenger
issue_28106: state: open
title: ENH Use Array API in mean_tweedie_deviance
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

xref #26024 

Inspired by #27904 

#### What does this implement/fix? Explain your changes.

Array API support in mean_tweedie_deviance


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: lithomas1
issue_28098: state: open
title: Adding (Mini-Batch) Spherical K-Means Clustering
body: ### Describe the workflow you want to enable

We need to use the spherical k-means algorithm to cluster text embeddings on a hypersphere. Currently there seem to be no reliable implementations available. Since we have a lot of input-data, we would like to implement the mini-batch version of the algorithm.

### Describe your proposed solution

Spherical k-means is a trivial modifications of normal k-means clustering where the centroids are projected onto the hypersphere in each step. Thus, we could add a boolean option `spherical=False` to the constructor of `MiniBatchKMeans` that toggles between standard k-means and spherical k-means. It is my understanding that we would then only have to l2-normalize the input data, l2-normalize the initial centroids here: https://github.com/scikit-learn/scikit-learn/blob/91d273ae892851ec3bdc4a21cffe163fbaed40f0/sklearn/cluster/_kmeans.py#L2128 and l2-normalize the updated centroids after this call: https://github.com/scikit-learn/scikit-learn/blob/91d273ae892851ec3bdc4a21cffe163fbaed40f0/sklearn/cluster/_kmeans.py#L2174 

### Describe alternatives you've considered, if relevant

There is the [spherecluster](https://github.com/jasonlaska/spherecluster) module, which has been broken since the release of scikit-learn 1.0.0 and is no longer maintained, therefore no viable alternative for users.

### Additional context

We are happy to prepare a PR for this ourselves!
user: Markus28
issue_28096: state: open
title: EFF Optimize memory usage for sparse matrices in LLE (Hessian, Modified and LTSA)
body: #### What does this implement/fix? Explain your changes.
This PR optimizes memory management with sparse matrices when using Modified Locally Linear Embedding.

Before this PR, a numpy NxN array was created, filled, and then converted to sparse. The creation of the said array can require huge memory when dealing with a large dataset.
On the dataset I was working with, the algorithm tried to allocate 400GB of RAM lol...

In the current PR, when M_sparse is true, the algorithm creates directly a sparse matrix, greatly reducing the memory requirements.

user: giorgioangel
issue_28093: state: open
title: Add verbose option to sklearn.inspection.permutation_importance()
body: ### Describe the workflow you want to enable

Add the verbose parameter to sklearn.inspection.permutation_importance.
This will improve the user experience by allowing them to evaluate how long the process will take, which is especially useful when working models with a high number of features.

### Describe your proposed solution

My solution would be to add "verbose" as a paramter in sklearn.inspection.permutation_importance and than add verbose inside the function like this:

```
scores = Parallel(n_jobs=n_jobs, verbose=verbose)(
        delayed(_calculate_permutation_scores)(
            estimator,
            X,
            y,
            sample_weight,
            col_idx,
            random_seed,
            n_repeats,
            scorer,
            max_samples,
        )
        for col_idx in range(X.shape[1])
    )
```



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

As someone who uses the library regularly, I was wondering if there's a reason why the 'verbose' parameter hasn't been implemented yet. I actually changed it locally and have been using it like this ever since!
user: philippolfert
issue_28084: state: open
title: DOC Update website to `pydata-sphinx-theme`
body: This issue is a continuation of [#26809](https://github.com/scikit-learn/scikit-learn/pull/26809) and aims to migrate the scikit-learn website towards [`pydata-sphinx-theme`](https://pydata-sphinx-theme.readthedocs.io/en/stable/). As this is an ambitious goal, this issue is to track the steps of the migration. cc @lucyleeow who guided me to open this issue.

#### Quick links

- [The `new_web_theme` branch](https://github.com/scikit-learn/scikit-learn/tree/new_web_theme)
- [Tracker for upstream issues](https://github.com/scikit-learn/scikit-learn/issues/28084#issuecomment-1893103722)
- Live preview: https://scikit-learn.org/_pst_preview

#### TODO before merging into `main`

- `doc-min-dependencies` is bypassed in CI for now and we need to reactivate it before merging into `main`. Also, we need to make sure all dependencies are documented, in particular `sphinx-design` which was not added in the very first setup PR. See also [#28379](https://github.com/scikit-learn/scikit-learn/pull/28379).
- Remove `themes/` and update `exclude_patterns` in `conf.py`: these are only useful for the old theme.
- Pin higher versions of `sphinx`, `pydata-sphinx-theme`, and `sphinx-gallery`. See also [tracker for upstream issues](https://github.com/scikit-learn/scikit-learn/issues/28084#issuecomment-1893103722).

#### Tracking work towards the `new_web_theme` branch

- [x] https://github.com/scikit-learn/scikit-learn/pull/28132 
- [x] https://github.com/scikit-learn/scikit-learn/pull/28331
- [x] https://github.com/scikit-learn/scikit-learn/pull/28336
- [ ] https://github.com/scikit-learn/scikit-learn/pull/28347
- [x] https://github.com/scikit-learn/scikit-learn/pull/28353
- [x] https://github.com/scikit-learn/scikit-learn/pull/28379
- [ ] https://github.com/scikit-learn/scikit-learn/pull/28401
- [x] https://github.com/scikit-learn/scikit-learn/pull/28408
- [x] https://github.com/scikit-learn/scikit-learn/pull/28409
- [x] https://github.com/scikit-learn/scikit-learn/pull/28415
- [ ] https://github.com/scikit-learn/scikit-learn/pull/28428
- [ ] https://github.com/scikit-learn/scikit-learn/pull/28448
- [ ] https://github.com/scikit-learn/scikit-learn/pull/28478
- [x] https://github.com/scikit-learn/scikit-learn/pull/28511
- [ ] https://github.com/scikit-learn/scikit-learn/pull/28512
- [ ] Tune testimonials and about pages (**Todo:** retain a similar styling; the logos do not look well in dark mode)
- [ ] Fix admonitions (**Todo:** make e.g. `.. note:: something` into `.. admonition:: something` with `:class: note`)

#### Tracking work related by towards the `main` branch

- [x] https://github.com/scikit-learn/scikit-learn/pull/28387
- [x] https://github.com/scikit-learn/scikit-learn/pull/28376
- [x] https://github.com/scikit-learn/scikit-learn/pull/28315
- [x] https://github.com/scikit-learn/scikit-learn/pull/28272
- [x] https://github.com/scikit-learn/scikit-learn/pull/28271
- [x] https://github.com/scikit-learn/scikit-learn/pull/28255
- [x] https://github.com/scikit-learn/scikit-learn/pull/28134
- [x] https://github.com/scikit-learn/scikit-learn/pull/28107
user: Charlie-XIAO
issue_28083: state: open
title: Reduce verbosity of error "Input X contains NaN" of PLSRegression
body: ### Describe the workflow you want to enable

Currently if PLSRegression (and other algorithms, judging by google) gets any NaNs as input, it raises this long multiline exception:
```
ValueError: Input X contains NaN. 
PLSRegression does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
```

(source is here: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/validation.py#L158)

I am running the algorithm regularly at fixed intervals and unattended, so I am getting very verbose logs from time to time. It would be nice to have an option to reduce verbosity of this error.

### Describe your proposed solution

Add an option to suppress long exception description and only emit "Input X contains NaN" message. Or only emit verbose error once, and then switch to short message.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: positron96
issue_28076: state: open
title: Partial Fix for Agglomerative clustering training error for seuclidean/mahalanobis affinity and single linkage #26961
body: #26961 Original issue [https://github.com/scikit-learn/scikit-learn/issues/26961](url)

#### What does this implement/fix? Explain your changes.

Following Mickey774 comments, there are now arguments V or VI in the AgglomerativeClustering class in order to forward a Variance matrix or a Covariance matrix used by Seuclidean or Mahalanobis metrics. Documentation was updated to deprecate the term "affinity" and only use the term "metric". If V=None or VI=None or nothing is passed as argument and metric='seuclidean' or metric='mahalanobis', there will be an auto computation of the Variance or Covariance matrix.

#### Any other comments?

Tested on Jupyterlab, PyTest passed.

Any comments or suggestions welcome :)
user: zeny07spartan
issue_28072: state: open
title: MNT Refactor fit method of NearestCentroid.
body: #### Reference Issues/PRs

None.

#### What does this implement/fix? Explain your changes.

This PR refactors the `fit` method of the NearestCentroid classifier.
Changes include:
- decoupling of the computation of class centroids and choosing the function for centroid computation
- improvements to References

#### Any other comments?

None.

user: fkdosilovic
issue_28065: state: open
title: DOC Combined examples for feature_selection.RFE and feature_selection.RFECV
body: #### Reference Issues/PRs

Follow up to #26950. Issue #26927

#### What does this implement/fix? Explain your changes.

This PR picks up from the work of @Shreesha3112 , whom I am very grateful to for providing the starting code.

I've followed the advice from @glemaitre in regards to this review: https://github.com/scikit-learn/scikit-learn/pull/26950#pullrequestreview-1712135224 . In particular, I've combined the RFE and RFECV examples into a single document. I've also swapped out the handwritten digits dataset for the breast cancer dataset, as the model performance for this particular dataset actually benefits from RFE.

I haven't deleted the redundant RFECV example, just in case.
user: raj-pulapakura
issue_28064: state: open
title: ENH make initial binning in HGBT parallel
body: #### Reference Issues/PRs
None

#### What does this implement/fix? Explain your changes.
This PR make the the finding of thresholds/quantiles in `_BinMapper.fit` parallel with `with concurrent.futures.ThreadPoolExecutor`, see https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor-example. This works indeed one of the recommended ways to make numpy parallel in pure Python, see https://numpy.org/doc/stable/reference/random/multithreading.html.

#### Any other comments?
Is there a reason, we never use a ThreadPoolExecutor?

The gain in execution speed is low as only a fraction of the fit time is spend finding the thresholds.
user: lorentzenchr
issue_28063: state: open
title: ENH add subsample to HGBT
body: #### Reference Issues/PRs
Fixes #16062 (#27139 is already merged). 


#### What does this implement/fix? Explain your changes.
Add `subsample` to `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor`. Similar to `subsample` in the old `GradientBoostingClassifier`.

#### Any other comments?
While the implementation is rather easy, suggestions for good tests are welcome.
user: lorentzenchr
issue_28059: state: open
title: ENH: Random Forest Classifier oob scaling/parallel
body: My team, working on a bioinformatics problem with high feature count (columns/dimensions in `X`), noticed that the `RandomForestClassifier` out of bag scoring doesn't scale with `n_jobs`. To be fair, `n_jobs` clearly says what it does support, though I do wonder if the out of bag predictions under the hood might also benefit from parallel support. Someone on my team seems to have found that it does help, but implemented externally to sklearn using the exposed base estimators. I suppose it might be nice to have that internally at some point, if there are no design reasons not to?

Sample reproducer code with latest stable release (`1.3.2`) on 16 cores/x86_64 Linux box (`i9-13900K`) is below the fold, and the scaling plot is underneath that. We also use far more estimators and features than that, so the delta is much greater, but the scaling trend is the main observation in any case.

<details>

```python
from time import perf_counter
import numpy as np
# sklearn 1.3.2
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt


timings_oob = []
timings_base = []
feature_counts = np.linspace(10, 180_000, 20, dtype=np.int64)

for feature_count in feature_counts:
    X, y = make_classification(n_samples=1_000,
                               n_features=feature_count,
                               random_state=0)
    for use_oob, timing_list in zip([True, False], [timings_oob, timings_base]):
        start = perf_counter()
        clf = RandomForestClassifier(n_estimators=50,
                                     random_state=0,
                                     oob_score=use_oob,
                                     n_jobs=16)
        clf.fit(X, y)
        timing_list.append(perf_counter() - start)

fig, ax = plt.subplots(1, 1)
ax.set_title(f"Random Forest OOB scaling performance")
ax.plot(feature_counts,
        timings_oob,
        label="WITH OOB",
        marker=".")
ax.plot(feature_counts,
        timings_base,
        label="NO oob",
        marker=".")
ax.set_xlabel("num features")
ax.set_ylabel("Time (s)")
ax.legend()
fig.savefig("bench_feat.png", dpi=300)
```

</details>

![image](https://github.com/scikit-learn/scikit-learn/assets/7903078/d07a173a-d847-4980-aa6b-02bc0cfc0f9a)

user: tylerjereddy
issue_28049: state: open
title: Plan for SGD and SAGA loss function migration
body: As a result of #15123, we now have a common private loss function module under `sklearn._loss`. In `sklearn.linear_models` we have 2 algorithms that need Cython version that calculate losses and gradients on single values (not on arrays), namely
- `_plain_sgd` as used in `SGDClassifier`, `SGDRegressor` and `SGDOneClassSVM`; and
- `sag_solver` as used in `LogisticRegression`.

My plan is to break this migration into smaller steps:
- [x] 1. #27979 Deprecate `loss_function_` attribute in v1.4 which gives access to the Cython loss functions that we want to replace.
- [x] 2. #27999 change order of Cython loss function arguments to align with the ones in `sklearn._loss`.
- [ ] 3. Carry out the deprecation after release 1.5 (to be released with 1.6)
- [ ] 4. #28029 Replace Cython losses with the ones from `sklearn._loss`, except multinomial one
- [ ] 5. #28037 Replace the multinomial loss


user: lorentzenchr
issue_28043: state: open
title: ENH Add support for np.nan values in SplineTransformer
body: #### Reference Issues/PRs
Closes #26793

#### What does this implement/fix? Explain your changes.
Adds support for np.nan values in SplineTransformer. 
- adds param `handle_missing : {'error', 'indicator'}` to __init__, where `error` preserves the previous behaviour and `indicator` handles nan values by setting their spline values to all 0s
- adds a masking column in case of nan values
- adds new test (but needs to be extended)

Yet to do:

1) How to handle the `use_sparse` cases in transform, because scipy `BSpline.build_matrix()` doesn't accept any nan (see CI). 

- So, should `sparse_output=True` and `handle_missing="indicator"` be prevented from being used together (and show an explicit error message) ,

- or passing x (column from X) into `BSpline.build_matrix()` with nans removed before (but that would implicitly change the user's intend without them noticing, add a warning(?)),

- or is there a solution which like in the dense case allows nan values to be calculated (which then are made into 0s)?

2) I believe in `_get_base_knot_positions` I have to prepare `_weighted_percentile` for excluding nan values similarity to how `np.nanpercentile` excludes nan values for the calculation of the base knots. I tried, but it was quite tricky. Edit: Just found that `np.nanpercentile` will have a sample_weight option soon: [PR 24254 in numpy](https://github.com/numpy/numpy/pull/24254)

3) Should an error also be raised in case the SplineTransformer was instantiated with (`handle_missing="error"`), then fitted without missing values and the X then contains missing values in transform?
user: StefanieSenger
issue_28038: state: open
title: ENH add zero_division in balanced_accuracy_score
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #26892

#### What does this implement/fix? Explain your changes.

This addresses an inconsistency in the balanced_accuracy_score function, where the calculated balanced accuracy was not equal to the macro-average recall score. The issue was traced to the absence of the handling of zero division which resulted in unexpected discrepancies. To rectify this, the implementation was modified to ensure that zero division is appropriately handled, and the adjusted balanced accuracy is consistent with the macro-average recall score. These changes guarantee that the balanced accuracy aligns with the expected behavior. The test suite was updated to reflect the corrected behavior and to ensure the accuracy of the metric in various scenarios.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: TengaCortal
issue_28037: state: open
title: MNT replace Cython loss functions in SGD part 3
body: #### Reference Issues/PRs
Follow-up of #28029 (which needs to be merged first), partially addresses https://github.com/scikit-learn/scikit-learn/issues/15123.

#### What does this implement/fix? Explain your changes.
This PR replaces the multinomial loss for SAGA.

#### Any other comments?
Only merge after release 1.5, i.e. this PR is to be released with v1.6.

user: lorentzenchr
issue_28036: state: open
title: add dbcv score (unsupervised clustering score) and one test comparing kmeans and dbscan dbcv score
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Towards #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Fixes #27259


#### What does this implement/fix? Explain your changes.

This adds the dbcv score which is a density based score for unsupervised clustering. I've also included one test that compares kmeans and dbscan's dbcv scores with the moon dataset. This follows [this paper](https://www.dbs.ifi.lmu.de/~zimek/publications/SDM2014/DBCV.pdf).

#### Any other comments?

I wasn't sure where to add all the helper functions for dbcv score as it is a bit more complex than other metrics so I added another file but feel free to propose other options as it is my first PR. The implementation follows closely [this repo](https://github.com/FelSiq/DBCV).

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Nylio-prog
labels: Needs Decision: Requires decision
Needs Decision - Include Feature: Requires decision regarding including feature
issue_28034: state: open
title: DOC Add note in Array API doc regarding support for devices without float 64 support
body: #### What does this implement/fix? Explain your changes.

Adds a note in the array api documentation that documents scikit-learn policy regarding support of devices that do not support float64 precision operations. (basically stating that it favors consistency with CPU behavior at the cost of data transfers to CPU,  over remaining on the device at the cost of capping compute to highest supported precision)

Discuted before with @betatim and @ogrisel , in particular during review of https://github.com/scikit-learn/scikit-learn/pull/27904

user: fcharras
issue_28029: state: open
title: ENH replace Cython loss functions in SGD part 2
body: #### Reference Issues/PRs
Follow-up of #27999 (which needs to be merged first). Partly addresses #15123.

#### What does this implement/fix? Explain your changes.
This PR replaces the Cython loss functions of SGD and SAGA with the ones from `_loss` (SquaredLoss, Huber, LogLoss) and inherits from `_loss._loss.CyLossFunction` for the remaining ones (Hinge, ..., and Multinomial).

Also, the loss functions form `sklearn.linear_model.__init__` are removed.

#### Any other comments?
Only merge after release 1.5, i.e. this PR is to be released with v1.6.
user: lorentzenchr
issue_27997: state: open
title: [MRG] [ENH] + [DOC] Adding ECCE-R and ECCE-MAD losses for calibration of classification models.
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
We are not aware of any ongoing PR for a similar implementation but please let us know if such PR exists. Also this is our first official PR to scikit-learn so please feel free to comment if any changes are needed. The main contributors of this PR are:

- Abel Sancarlos Gonzalez @Abel-24pal
- Edgar Bahilo Rodriguez @edgBR

#### What does this implement/fix?  

The lack of a metric in scikit-learn to assess just the calibration performance of a model.  

To this end, we propose the enhancement of the scikit-learn library adding the best two metrics to evaluate the calibration of a model that have been mathematically proved. These two metrics are: 

- ECCE-MAD: ecce_mad_loss.
- ECCE-R: ecce_r_loss. 

We can see growing interest in the use of these metrics in the community (available in the references below). Adding these metrics would be an enhancement to scikit-learn that would have a positive impact on many of the industries that use scikit-learn. 

#### What are we adding in the PR? 

1. The metrics ecce_mad_loss and ecce_r_loss. 
2. The scorers to be used in cross validation objects. 
3. The corresponding API documentation. 
4. The test that the functions work. 
5. An example of use. 
6. User guide information about them: mathematical definition, use, references, etc. 

#### Detailed technical explanation 

Reliability diagrams are an extremely useful visual tool. However, reliable metrics to assess calibration are also highly demanded in industry, for instance, for an automatic model selection based on this assessment. We propose an enhancement where scikit-learn provides the ECCE metrics to assess calibraiton (ecce_mad_loss and the ecce_r_loss).  

The following advantages motivate the choice of the ECCEs (empirical or estimated cumulative calibration errors) over other metrics based on binning. First, ECCEs yield trustworthy results needing a lower number of observations than the metrics based on binning such as the ECE metrics. Furthermore, ECCE metrics do not need to set any parameters such as the number of bins removing the trade-off between statistical confidence and power for the methodologies based on binning. Moreover, choosing among the possible binnings can be confusing, yet makes all the difference [1]. This does ECCE metrics more convenient and robust for an automatic solution. In addition, alternative metrics such as the Brier score could be used to assess how well a classifier is calibrated, but they come with limitations and drawbacks. In fact, a lower Brier score loss does not necessarily mean a better calibration, thus not being appropriate if your main goal is to find the best probability estimator [2] [3] [4].  

[1]  Imanol Arrieta-Ibarra, Paman Gujral, Jonathan Tannen, Mark Tygert, and Cherie Xu., Metrics of calibration for probabilistic predictions, J. Mach. Learn. Res. Vol. 23. (2022) 

[2] Bella, Ferri, Hernández-Orallo, and Ramírez-Quintana “Calibration of Machine Learning Models” in Khosrow-Pour, M. “Machine learning: concepts, methodologies, tools and applications.” Hershey, PA: Information Science Reference (2012). 

[3] Flach, Peter, and Edson Matsubara. “On classification, ranking, and probability estimation.” Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum fr Informatik (2008). 

[4] Abel Sancarlos, Edgar Bahilo, Pablo Mozo, Lukas Norman, Obaid Ur Rehma and Mihails Anufrijevs, 
“Towards a data-driven debt collection strategy based on an advanced machine learning framework.”, (2023) 
user: edgBR
issue_27993: state: open
title: [RFC] Allow handling of NaNs in multi-task Random Forests
body: ### Describe the workflow you want to enable

 Currently the RFR implementation is only capable of handling dense multi-task problems is there any scope to change the underlying algorithm to handle NaNs as a special case or would this break the API given that it might extend to allowing NaNs in the single class case? 

The hypothesis is that I may benefit from knowing the information about the available labels in the multi-task model even though I am missing labels for some tasks.

### Describe your proposed solution

boolean flag in the constructor that allows NaNs to be ignored when calculating the impurity.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

```python
>>> import numpy as np
>>> from sklearn.ensemble import RandomForestRegressor
>>> rfr=RandomForestRegressor()
>>> X = np.random.randn(5,10)
>>> y = np.random.randn(5,2)
>>> y[4,1]=np.nan
>>> rfr.fit(X, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 1151, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/ensemble/_forest.py", line 348, in fit
    X, y = self._validate_data(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 621, in _validate_data
    X, y = check_X_y(X, y, **check_params)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1163, in check_X_y
    y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1173, in _check_y
    y = check_array(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 959, in check_array
    _assert_all_finite(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 124, in _assert_all_finite
    _assert_all_finite_element_wise(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 173, in _assert_all_finite_element_wise
    raise ValueError(msg_err)
ValueError: Input y contains NaN.
```
user: CompRhys
issue_27982: state: open
title: Ensure that we have an example in the docstring of each public function or class
body: We should make sure that we have a small example for all public functions or classes. Most of the missing examples are linked to functions.

I could list the following classes and functions for which `numpydoc` did not find any example:

- [x] sklearn.base.BaseEstimator
- [x] sklearn.base.BiclusterMixin
- [x] sklearn.base.ClassNamePrefixFeaturesOutMixin
- [x] sklearn.base.ClassifierMixin
- [x] sklearn.base.ClusterMixin
- [x] sklearn.base.DensityMixin
- [x] sklearn.base.MetaEstimatorMixin
- [x] sklearn.base.OneToOneFeatureMixin
- [x] sklearn.base.OutlierMixin
- [x] sklearn.base.RegressorMixin
- [x] sklearn.base.TransformerMixin
- [x] sklearn.base.clone
- [x] sklearn.base.is_classifier
- [x] sklearn.base.is_regressor
- [x] sklearn.cluster.affinity_propagation
- [x] sklearn.cluster.cluster_optics_dbscan
- [x] sklearn.cluster.cluster_optics_xi
- [x] sklearn.cluster.compute_optics_graph
- [x] sklearn.cluster.estimate_bandwidth
- [x] sklearn.cluster.k_means
- [x] sklearn.cluster.mean_shift
- [x] sklearn.cluster.spectral_clustering
- [x] sklearn.cluster.ward_tree
- [x] sklearn.covariance.graphical_lasso
- [x] sklearn.covariance.ledoit_wolf
- [x] sklearn.covariance.ledoit_wolf_shrinkage
- [x] sklearn.covariance.shrunk_covariance
- [x] sklearn.datasets.clear_data_home
- [x] sklearn.datasets.dump_svmlight_file
- [ ] sklearn.datasets.fetch_20newsgroups
- [ ] sklearn.datasets.fetch_20newsgroups_vectorized
- [x] sklearn.datasets.fetch_california_housing
- [x] sklearn.datasets.fetch_covtype
- [ ] sklearn.datasets.fetch_kddcup99
- [ ] sklearn.datasets.fetch_lfw_pairs
- [ ] sklearn.datasets.fetch_lfw_people
- [ ] sklearn.datasets.fetch_olivetti_faces
- [x] sklearn.datasets.fetch_openml
- [ ] sklearn.datasets.fetch_rcv1
- [x] sklearn.datasets.fetch_species_distributions
- [ ] sklearn.datasets.get_data_home
- [x] sklearn.datasets.load_diabetes
- [x] sklearn.datasets.load_files
- [ ] sklearn.datasets.load_linnerud
- [ ] sklearn.datasets.load_svmlight_files
- [ ] sklearn.datasets.make_biclusters
- [x] sklearn.datasets.make_checkerboard
- [x] sklearn.datasets.make_circles
- [x] sklearn.datasets.make_classification
- [x] sklearn.datasets.make_friedman1
- [x] sklearn.datasets.make_friedman2
- [x] sklearn.datasets.make_friedman3
- [x] sklearn.datasets.make_gaussian_quantiles
- [ ] sklearn.datasets.make_hastie_10_2
- [ ] sklearn.datasets.make_low_rank_matrix
- [ ] sklearn.datasets.make_moons
- [x] sklearn.datasets.make_multilabel_classification
- [ ] sklearn.datasets.make_s_curve
- [ ] sklearn.datasets.make_sparse_coded_signal
- [x] sklearn.datasets.make_sparse_spd_matrix
- [ ] sklearn.datasets.make_sparse_uncorrelated
- [x] sklearn.datasets.make_spd_matrix
- [ ] sklearn.datasets.make_swiss_roll
- [x] sklearn.decomposition.dict_learning
- [x] sklearn.decomposition.dict_learning_online
- [x] sklearn.decomposition.sparse_encode
- [x] sklearn.feature_extraction.image.grid_to_graph
- [ ] sklearn.feature_extraction.image.img_to_graph
- [x] sklearn.feature_extraction.image.reconstruct_from_patches_2d
- [x] sklearn.feature_selection.SelectorMixin
- [x] sklearn.feature_selection.chi2
- [x] sklearn.feature_selection.f_classif
- [x] sklearn.feature_selection.f_regression
- [x] sklearn.feature_selection.mutual_info_classif
- [x] sklearn.feature_selection.mutual_info_regression
- [x] sklearn.feature_selection.r_regression
- [x] sklearn.gaussian_process.kernels.Kernel
- [x] sklearn.get_config
- [x] sklearn.isotonic.check_increasing
- [x] sklearn.isotonic.isotonic_regression
- [ ] sklearn.linear_model.enet_path
- [ ] sklearn.linear_model.lars_path
- [ ] sklearn.linear_model.lars_path_gram
- [ ] sklearn.linear_model.orthogonal_mp
- [ ] sklearn.linear_model.orthogonal_mp_gram
- [x] sklearn.linear_model.ridge_regression
- [x] sklearn.manifold.locally_linear_embedding
- [x] sklearn.manifold.smacof
- [x] sklearn.manifold.spectral_embedding
- [x] sklearn.manifold.trustworthiness
- [x] sklearn.metrics.calinski_harabasz_score
- [x] sklearn.metrics.check_scoring
- [x] sklearn.metrics.cohen_kappa_score
- [x] sklearn.metrics.consensus_score
- [x] sklearn.metrics.coverage_error
- [x] sklearn.metrics.davies_bouldin_score
- [x] sklearn.metrics.get_scorer
- [x] sklearn.metrics.get_scorer_names
- [x] sklearn.metrics.homogeneity_completeness_v_measure
- [x] sklearn.metrics.label_ranking_loss
- [x] sklearn.metrics.mutual_info_score
- [x] sklearn.metrics.pairwise.additive_chi2_kernel
- [x] sklearn.metrics.pairwise.chi2_kernel
- [x] sklearn.metrics.pairwise.cosine_distances
- [x] sklearn.metrics.pairwise.cosine_similarity
- [x] sklearn.metrics.pairwise.distance_metrics
- [x] sklearn.metrics.pairwise.kernel_metrics
- [x] sklearn.metrics.pairwise.laplacian_kernel
- [x] sklearn.metrics.pairwise.linear_kernel
- [x] sklearn.metrics.pairwise.paired_cosine_distances
- [x] sklearn.metrics.pairwise.paired_euclidean_distances
- [x] sklearn.metrics.pairwise.pairwise_kernels
- [x] sklearn.metrics.pairwise.polynomial_kernel
- [x] sklearn.metrics.pairwise.rbf_kernel
- [x] sklearn.metrics.pairwise.sigmoid_kernel
- [x] sklearn.metrics.pairwise_distances
- [x] sklearn.metrics.pairwise_distances_argmin
- [x] sklearn.metrics.pairwise_distances_argmin_min
- [x] sklearn.metrics.silhouette_samples
- [x] sklearn.metrics.silhouette_score
- [x] sklearn.model_selection.check_cv
- [x] sklearn.model_selection.permutation_test_score
- [x] sklearn.model_selection.validation_curve
- [x] sklearn.neighbors.sort_graph_by_row_values
- [x] sklearn.preprocessing.binarize
- [x] sklearn.preprocessing.maxabs_scale
- [x] sklearn.preprocessing.minmax_scale
- [x] sklearn.preprocessing.normalize
- [x] sklearn.preprocessing.robust_scale
- [x] sklearn.preprocessing.scale
- [x] sklearn.set_config
- [x] sklearn.show_versions
- [x] sklearn.svm.l1_min_c
- [x] sklearn.utils._safe_indexing
- [x] sklearn.utils.arrayfuncs.min_pos
- [x] sklearn.utils.as_float_array
- [x] sklearn.utils.assert_all_finite
- [x] sklearn.utils.check_X_y
- [x] sklearn.utils.check_array
- [x] sklearn.utils.check_consistent_length
- [x] sklearn.utils.check_random_state
- [x] sklearn.utils.check_scalar
- [x] sklearn.utils.class_weight.compute_class_weight
- [x] sklearn.utils.class_weight.compute_sample_weight
- [x] sklearn.utils.deprecated
- [x] sklearn.utils.discovery.all_displays
- [x] sklearn.utils.discovery.all_estimators
- [x] sklearn.utils.discovery.all_functions
- [x] sklearn.utils.estimator_checks.check_estimator
- [x] sklearn.utils.estimator_html_repr
- [x] sklearn.utils.extmath.density
- [x] sklearn.utils.extmath.randomized_range_finder
- [x] sklearn.utils.extmath.safe_sparse_dot
- [x] sklearn.utils.indexable
- [x] sklearn.utils.metadata_routing.MetadataRequest
- [x] sklearn.utils.metadata_routing.MetadataRouter
- [x] sklearn.utils.metadata_routing.MethodMapping
- [x] sklearn.utils.metadata_routing.get_routing_for_object
- [x] sklearn.utils.metadata_routing.process_routing
- [x] sklearn.utils.murmurhash3_32
- [x] sklearn.utils.parallel.Parallel
- [x] sklearn.utils.parallel.delayed
- [x] sklearn.utils.parallel_backend
- [x] sklearn.utils.random.sample_without_replacement
- [x] sklearn.utils.register_parallel_backend
- [x] sklearn.utils.safe_mask
- [x] sklearn.utils.safe_sqr
- [x] sklearn.utils.sparsefuncs.incr_mean_variance_axis
- [x] sklearn.utils.sparsefuncs.inplace_column_scale
- [x] sklearn.utils.sparsefuncs.inplace_csr_column_scale
- [x] sklearn.utils.sparsefuncs.inplace_row_scale
- [x] sklearn.utils.sparsefuncs.inplace_swap_column
- [x] sklearn.utils.sparsefuncs.inplace_swap_row
- [x] sklearn.utils.sparsefuncs.mean_variance_axis
- [ ] sklearn.utils.sparsefuncs_fast.inplace_csr_row_normalize_l1
- [ ] sklearn.utils.sparsefuncs_fast.inplace_csr_row_normalize_l2
- [x] sklearn.utils.validation.check_is_fitted
- [x] sklearn.utils.validation.check_memory
- [x] sklearn.utils.validation.check_symmetric
- [x] sklearn.utils.validation.column_or_1d

The code used to find the list above is detailed below:

<details>

```python
import importlib
import inspect
from pathlib import Path

from numpydoc.docscrape import NumpyDocString

path_sklearn_doc = Path(
    "/{path_to_git_repo}/scikit-learn/doc/_build/html/stable/"
    "modules/generated"
)

missing_examples_name = []
for document in path_sklearn_doc.glob("*.html"):
    extracted_doc = []
    full_name = document.stem
    try:
        module_name, class_or_function_name = full_name.rsplit(".", maxsplit=1)
        module = importlib.import_module(module_name)
        class_or_function = getattr(module, class_or_function_name)
    except (ValueError, AttributeError, ImportError):
        # This is due to the experimental module and function with
        # module name
        continue
    is_class = inspect.isclass(class_or_function)
    docstring = NumpyDocString(class_or_function.__doc__)
    if not docstring["Examples"]:
        missing_examples_name.append(full_name)

for full_name in sorted(missing_examples_name):
    print(f"- [ ] {full_name}")
```

</details>
user: glemaitre
labels: good first issue: Easy with clear instructions to resolve
issue_27980: state: open
title: support `handle_unknown = infrequent_if_exist`  and `handle_unknown = ignore` in OrdinalEncoder
body: Hello thanks for this amazing project!  This PR tries to support the options `handle_unknown = infrequent_if_exist`  and `handle_unknown = ignore`  in the `OrdinalEncoder`.

#### Reference Issues/PRs

https://github.com/scikit-learn/scikit-learn/issues/27629

#### What does this implement/fix? Explain your changes.

Adds to the `OrdinalEncoder` the arguments `infrequent_if_exist` and `ignore` similar than `OneHotEnconder` --- When `ignore` is set or `ifrequent_if_exist` but infrequent is not enabled, all the unknowns are mapped to -1, this value seemed natural to me when working with ordinal values. 

Also i modified a bit the warning used to notify unknowns values, i don't know why it was tied to `self.drop`


user: andresliszt
issue_27977: state: open
title: Routing metadata to the `response_method` used by a scorer
body: ### Describe the workflow you want to enable

I would like to pass sample properties to the response method (eg `predict`) called by a scorer.
For example, the `fairlearn` package has a `ThresholdOptimizer` estimator which needs (in addition to X and y) the `sensitive_features` argument both for fit and predict.

AFAICT I can pass arguments to the score function (the metric), but not to the response method of the estimator.

```python
import numpy as np
import sklearn
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, make_scorer
from fairlearn.postprocessing import ThresholdOptimizer
from fairlearn.metrics import demographic_parity_difference


sklearn.set_config(enable_metadata_routing=True)

rng = np.random.default_rng(0)
X = rng.normal(size=(10, 3))
y = rng.integers(0, 2, size=X.shape[0])
sensitive = rng.integers(0, 2, size=X.shape[0])

classifier = (
    ThresholdOptimizer(estimator=DummyClassifier(), predict_method="auto")
    .set_fit_request(sensitive_features=True)
    .set_predict_request(sensitive_features=True)
    .fit(X, y, sensitive_features=sensitive)
)

scoring = make_scorer(accuracy_score)
scoring(classifier, X, y, sensitive_features=sensitive) # TypeError: predict() missing 1 argument -- how could I pass `sensitive_features to predict() ?

# passing arguments to the score function (demographic_parity_difference) is OK
classifier = DummyClassifier().fit(X, y)
scoring = make_scorer(
    demographic_parity_difference, greater_is_better=False
).set_score_request(sensitive_features=True)

scoring(classifier, X, y, sensitive_features=sensitive)

```

This also applies when using a scorer indirectly, for example in `cross_validate`

### Describe your proposed solution

Maybe the scorers could have a method like `set_predict_request` or `set_response_request` to specify which parameters should be forwarded to the response method?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

https://fairlearn.org/v0.9/api_reference/generated/fairlearn.postprocessing.ThresholdOptimizer.html
https://fairlearn.org/v0.9/api_reference/generated/fairlearn.metrics.demographic_parity_difference.html
user: jeromedockes
issue_27973: state: open
title: Bug in utils/multiclass.py/_ovr_decision_function
body: ### Describe the workflow you want to enable

Dear scikit learn developpers,

I think the implementation of `_ovr_decision_function` in utils
/multiclass.py doesn't work properly when the parameter `confidences` is probability. While as the documentation suggests, it can be a probability .

```
confidences : array-like of shape (n_samples, n_classifiers)
        Decision functions or predicted probabilities for positive class
        for each binary classifier.
```

The problem is the following two lines of codes

```
sum_of_confidences[:, i] -= confidences[:, k]
sum_of_confidences[:, j] += confidences[:, k]
```

In this context, there is a binary classifier for class `i` vs `j`. And `j` is the positive class. 

If `confidences` is "decision_function", then it works. Because if "decision funtion" is negative, it means the classifier thinks the negatve class `i` is more possible. And the `-=` will increase the `sum_of_confidences` of `i`, and decrease the `sum_of_confidences` of `j`.

However, if  `confidences` is "probability", it doesn't work. Because probability is always greater than zero. So the `sum_of_confidences` of `i` will always decrease, even when `i` is more likely to happend (prob of `j` < 0.5).




### Describe your proposed solution

"decision_function" is centered at 0, while "probability" is centerd at 0.5. These two cases should be handled seperately.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: newyorker123
issue_27972: state: open
title: Is the time complexity of neural network in the doc right?
body: ### Describe the issue linked to the documentation

Are you sure the [time complexity](https://scikit-learn.org/stable/modules/neural_networks_supervised.html#complexity) is right? Exponential complexity with respect to the number of layers rather than polynomial?
![image](https://github.com/scikit-learn/scikit-learn/assets/47685165/e183f52f-f03e-41d8-8192-74e9a410faf5)



### Suggest a potential alternative/fix

I notice a different answer from [here](https://ai.stackexchange.com/questions/5728/what-is-the-time-complexity-for-training-a-neural-network-using-back-propagation/20281?newreg=92fccd4d6b51442db4e6d1dcc1dcfccf), and I think it right.
user: BraveDrXuTF
issue_27971: state: open
title: FIX `KernelDensity` incorrectly handling bandwidth
body: Towards #25623, #26658.

See [this gist](https://gist.github.com/Charlie-XIAO/cac9eb1260d73bc9e0b9aea67cd38d7a) for some results of this PR. Without `sample_weight` the results are now consistent with `scipy` implementation, but with `sample_weight` it is failing. In particular, *adding an integer weight* is not equivalent to *repeating the sample that many times* with this PR. I failed to find a good reason why it fails with `sample_weight` (maybe I missed something), could you help if possible @ogrisel? Also @thomasjpfan and @glemaitre who were involved in #25623 for help.

Also, I made some changes in the binary tree code, see #27186, because I think that part is wrong. Seems that the change does not cause any test to fail, neither does it solve the `sample_weight` problem in this PR.

By the way, though not related to this PR, the implementation of weighted KDE in scikit-learn seems to be very slow (#10803). It needs to traverse all points in a node and sum their weights up every time, which makes the tree implementation (that should be fast) several times slower even than the naive implementation of `scipy` as data size scales up.
user: Charlie-XIAO
issue_27967: state: open
title: ⚠️ CI failed on Linux.pylatest_pip_openblas_pandas ⚠️
body: **CI is still failing on [Linux.pylatest_pip_openblas_pandas](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=64174&view=logs&j=78a0bf4f-79e5-5387-94ec-13e67d216d6e)** (Feb 20, 2024)
- test_minibatch_sensible_reassign[34]
user: scikit-learn-bot
issue_27966: state: open
title: [ENH] Add missing-value support for ExtaTreeClassifier and ExtaTreeRegressor
body: #### Reference Issues/PRs
Towards: #27931 

Follow-up to: https://github.com/scikit-learn/scikit-learn/pull/26391 and https://github.com/scikit-learn/scikit-learn/pull/23595

#### What does this implement/fix? Explain your changes.
- Adds missing-value support to `RandomSplitter` in _splitter.pyx
- Enables the "random" splitter kwarg for ExtraTreeClassifier  and ExtraTreeRegressor
- Adds unit-tests for ExtraTreeClassifier  and ExtraTreeRegressor


#### Any other comments?
Compared to BestSplitter, there can be an expected cost to doing splits on missing-values, as we can either:

1. choose a random threshold and then randomly send missing-values to left/right, OR
2. send all missing-values to left/right and all non-missing values to right/left

This can be done randomly, OR the second option can actually be evaluated. The tradeoff imo comes from the assumption of the missing-values:

- if missing-completely-at-random (MCAR), then option 1 is ideal because one simply should ignore the missing values or impute them
- if missing-at-random (MAR), then option 2 is nice because sometimes the missing-ness of the data can be informative.

However, I think the difference at a tree level is not super important. E.g. in the ExtraTree forest, https://github.com/scikit-learn/scikit-learn/pull/28268 demonstrates that the ExtraTrees when combined as a forest are resilient and predictive for missing-values.

## Benchmarks

There is some complexity involved in [checking](https://github.com/scikit-learn/scikit-learn/blob/64ad1cd8cf6b1e571fd9dcb1ee0c33144958ece5/sklearn/tree/_classes.py#L217-L229) if there are missing values. However, this only occurs at the Python level as shown by the following [benchmark](https://gist.github.com/adam2392/964a2b9da09525c23068324eb6fc72ee). In terms of the Cython code, there is no regression.

<details>
<summary>Benchmarks with and without Python Check</summary>

![results_image_without_pythoncheck](https://github.com/scikit-learn/scikit-learn/assets/3460267/58899904-430c-422f-b98f-e922384759df)
![results_image](https://github.com/scikit-learn/scikit-learn/assets/3460267/268eb920-7e6b-4b62-a9cf-d0919736261c)

</details>

Also ran this [benchmark](https://gist.github.com/adam2392/f283d5e6a2e8408a6f187a3079990674) for ExtraTrees, which demonstrates that this check is negligible at the forest level, since it only occurs once. See https://github.com/scikit-learn/scikit-learn/pull/28268, which has the short code to enable it for ExtraTrees.

<details>
<summary>Benchmarks on ExtraTrees</summary>

![results_image](https://github.com/scikit-learn/scikit-learn/assets/3460267/1c6cc01f-6479-4249-8fb7-5ab5aaf72c18)

</details>


user: adam2392
issue_27964: state: open
title: Correct scale back for PLS regression coefficients 
body: ### Describe the bug

In `cross_decomposition/_pls.py`, PLS regression coefficients are calculated in class `_PLS` (starts at line 165). In this class, when `scale=True`, data are scaled (on line 265). In that case, the resulting regression coefficients need to be scaled back to the original scale, such that they represent the relationship between the original X and y. The way this scale back is done on line 360, is wrong: https://github.com/scikit-learn/scikit-learn/blob/3f89022fa04d293152f1d32fbc2a5bdaaf2df364/sklearn/cross_decomposition/_pls.py#L360
This is wrong because rescaling is done by only adjusting for the `y_std`. 

The correct formula is to scale back as `self.coef_ = (self.coef_ * self._y_std/self.x_std).T`. It is easy to verify the latter for ordinary least squares regression as they will match exactly. As PLS is only rotationally invariant, the rescaled coefficients from this proposal will not match exactly, but they will be much closer to coefficients from unscaled data than the present version. Example code given below.

### Steps/Code to Reproduce

"""
import numpy as np 
from sklearn.linear_model import LinearRegression
from direpack import VersatileScaler
from sklearn.cross_decomposition import PLSRegression
"""

Simulate Data
"""
X = np.random.multivariate_normal(np.zeros(3),np.diag(np.ones(3)),500)
YL = np.dot(X,np.array([3,-3,5]).reshape((-1,1))) + np.random.multivariate_normal(np.zeros(3),np.diag(np.ones(3)/100),500)
""""

Test both options for OLS regression 
"""
LRns = LinearRegression()
LRns.fit(X,YL)
LRns.coef_
LRs = LinearRegression()
LRs.fit(Xs,YLs)
LRs.coef_
"""
`LRs.coef_` and `LRns.coef_` are identical 

Now test both options for PLS using internal scaling
"""
PLSns = PLSRegression(n_components=2, scale=False)
PLSns.fit(X,YL)
PLSns.coef_
PLSms = PLSRegression(n_components=2, scale=False)
PLSms.fit(Xs,YLs)
PLSms.coef_
""" 
Off! 

Now the proposed solution 
"""
PLSms = PLSRegression(n_components=2, scale=False)
PLSms.fit(Xs,YLs)
PLSms.coef_ 
PLSms.coef_ *sY/sX
"""
Much closer !!

### Expected Results

Rescaled regression coefficients much closer to coefficients calculated from unscaled data 

### Actual Results

Actual coefficients are off. One example running the code above:

- Coefficients from unscaled data : 
array([[ 2.74182398, -3.06332548,  5.08010417],
       [ 2.73919939, -3.0741624 ,  5.09501747],
       [ 2.7424613 , -3.06724517,  5.08589201]])

- Coefficients using the `scale=True` rescaling option: 
array([[ 2.74182398, -3.06332548,  5.08010417],
       [ 2.73919939, -3.0741624 ,  5.09501747],
       [ 2.7424613 , -3.06724517,  5.08589201]])
--> OFF 

- Coefficients that would be obtained from proposed solution: 
array([[ 3.0012696 , -3.00657879,  4.9926381 ],
       [ 2.99094369, -3.00971523,  4.99484826],
       [ 2.99880995, -3.00725971,  4.99306938]])
--> much closer! 

### Versions

```shell
System:
    python: 3.11.4 | packaged by Anaconda, Inc. | (main, Jul  5 2023, 13:38:37) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\serneels\AppData\Local\anaconda3\python.exe
   machine: Windows-10-10.0.19044-SP0

Python dependencies:
      sklearn: 1.3.0
          pip: 23.2.1
   setuptools: 68.0.0
        numpy: 1.24.3
        scipy: 1.10.1
       Cython: 0.29.36
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 2.2.0

Built with OpenMP: True
```

user: SvenSerneels
labels: Needs Triage: Issue requires triage
issue_27963: state: open
title: ENH Simplify pytest global random test plugin
body: #### Reference Issues/PRs

Fix https://github.com/scikit-learn/scikit-learn/issues/27806

@Charlie-XIAO can you double-check this fixes it for you? This fixes it in @fcharras Windows VM and mine too, but one more confirmation would be nice.

#### What does this implement/fix? Explain your changes.

I think we were in a weird edge-case of pytest by combining registering a plugin in the `sklearn` tree calling it via setup.cfg and sometimes registering inside `conftest.py` some reason. It seems like by moving all the code to `conftest.py` we avoid the issue.

[pytest_generate_tests](https://docs.pytest.org/en/latest/example/parametrize.html#generating-parameters-combinations-depending-on-command-line) seems a slightly simpler way to parameterize the global_random_seeds tests based on the variable SKLEARN_TESTS_GLOBAL_RANDOM_SEED.

cc @jeremiedbb who was involved in the global random seed plugin originally IIRC https://github.com/scikit-learn/scikit-learn/pull/22749. **Edit**: actually looks like @ogrisel was the one who created the global random seed plugin.
user: lesteve
issue_27961: state: open
title: [WIP] add support for array API to RidgeCV
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

This PR extends the one for Ridge (still WIP, #27800) to use the array API in `RidgeCV` and `RidgeClassifierCV` (when cv="gcv")

#### What does this implement/fix? Explain your changes.

this could make those estimators faster as an important part of their computational cost is due to compute either an eigendecomposition of XX^T or an SVD of X

#### Any other comments?

The `_RidgeGCV` has numerical precision issues when computations are done in float32, which is why ATM in the main branch it always uses [float64](https://github.com/scikit-learn/scikit-learn/blob/8f5ff3978fa9a6cc27868a30f22d5c12f0f59d03/sklearn/linear_model/_ridge.py#L1982)
I'm not sure what should be done for array API inputs on devices that do not have float64

not handled yet:
- [ ] RidgeClassifierCV

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: jeromedockes
issue_27957: state: open
title: Standard "Total Variance" Scaler
body: ### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)
user: rebeccaherman1
issue_27948: state: open
title: Pairwise distances (single precision) throwing seg fault on AWS c6i.metal instances
body: ### Describe the bug

## Pairwise distances (single precision) throwing seg fault on AWS c6i.metal instances

### The Issue

Applying pairwise (Euclidean) distances on a matrix of size 5000x5000.

```python
import numpy as np
from sklearn.metrics import pairwise_distances

samples = 5000
features = 5000
x = np.random.RandomState(0).random_sample((samples,features))
x = x.astype(np.float32, copy=False)  # to convert from fl64 -> fl32
dist = pairwise_distances(x, metric='euclidean', n_jobs=-1)
```
The instance c6i.metal has 128 vCPUs.

When `n_jobs=1` the script runs fine.  But when `n_jobs` is larger than 67, I get a warning saying

```
OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata.
To avoid this warning, please rebuild your copy of OpenBLAS with a larger NUM_THREADS setting
or set the environment variable OPENBLAS_NUM_THREADS to 128 or lower
```

(for float64, same warning is thrown only when `n_jobs` is larger than 68).

When `n_jobs=-1` (or 128), I get the same warning for float64. But for float32, the script does not exit successfully and throws a segmentation fault along with a larger error message (note that there is no seg fault for double precision float).
```
OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata.
To avoid this warning, please rebuild your copy of OpenBLAS with a larger NUM_THREADS setting
or set the environment variable OPENBLAS_NUM_THREADS to 128 or lower
OpenBLAS : Program is Terminated. Because you tried to allocate too many memory regions.
This library was built to support a maximum of 128 threads - either rebuild OpenBLAS
with a larger NUM_THREADS value or set the environment variable OPENBLAS_NUM_THREADS to
a sufficiently small number. This error typically occurs when the software that relies on
OpenBLAS calls BLAS functions from many threads in parallel, or when your computer has more
cpu cores than what OpenBLAS was configured to handle.
```


Setting `OPENBLAS_NUM_THREADS` to 128, as mentioned in the error message has no effect. For very low values of `OPENBLAS_NUM_THREADS`, the execution gets complete but throws the `NUM_THREADS` exceeded message, unless it is set to 1.


### Environment Used

- **Machine**: I am using AWS c6i.metal instances, which use Intel Xeon Platinum 8375 processors.
There are 2 sockets each containing 64 vCPUs. 

- **OS**: Ubuntu 22.04.2 LTS

- **Python**: 3.10.12

`scikit-learn` was installed using pip (v22.0.2)  on a freshly created virtual
envirnonment created using `venv` module.

- **scikit-learn**: 1.3.2

- **numpy**: 1.26.2

- **scipy**: 1.11.4


### How to replicate

Run the script given above on a c6i.metal instance (or some other intel CPU with
larger number of cores maybe?). Before running:

1. Create virtual environment: `python3 -m venv new-env`.
2. Activate the environment: `source ./new-env/bin/activate`.
3. Install scikit-learn: `pip install scikit-learn`.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.metrics import pairwise_distances

samples = 5000
features = 5000
x = np.random.RandomState(0).random_sample((samples,features))
x = x.astype(np.float32, copy=False)  # to convert from fl64 -> fl32
dist = pairwise_distances(x, metric='euclidean', n_jobs=-1)
```

### Expected Results

No error, no warnings or segmentation faults.

### Actual Results

```
OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata.
To avoid this warning, please rebuild your copy of OpenBLAS with a larger NUM_THREADS setting
or set the environment variable OPENBLAS_NUM_THREADS to 128 or lower
OpenBLAS : Program is Terminated. Because you tried to allocate too many memory regions.
This library was built to support a maximum of 128 threads - either rebuild OpenBLAS
with a larger NUM_THREADS value or set the environment variable OPENBLAS_NUM_THREADS to
a sufficiently small number. This error typically occurs when the software that relies on
OpenBLAS calls BLAS functions from many threads in parallel, or when your computer has more
cpu cores than what OpenBLAS was configured to handle.
Segmentation fault (core dumped)
```

### Versions

```shell
Python dependencies:
      sklearn: 1.3.2
          pip: 22.0.2
   setuptools: 59.6.0
        numpy: 1.26.2
        scipy: 1.11.4
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 128
         prefix: libgomp
       filepath: /home/dhanus/Projects/issues-fujitsu/pairwise-distances/new/lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 64
         prefix: libopenblas
       filepath: /home/dhanus/Projects/issues-fujitsu/pairwise-distances/new/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: blas
   internal_api: openblas
    num_threads: 64
         prefix: libopenblas
       filepath: /home/dhanus/Projects/issues-fujitsu/pairwise-distances/new/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: SkylakeX
```

user: DhanusML
issue_27947: state: open
title: Allowing to group infrequent categories in `HistGradientBoosting`
body: ### Describe the workflow you want to enable

`HistGradientBoostingClassifier` and `HistGradientBoostingRegressor` have built-in support for categorical features and use an `OrdinalEncoder` to encode them. Each feature must have less than `max_bins` (255) categories and if there are more we get a `ValueError`.

Would it be useful to add a parameter allowing to group together the least frequent categories when there are too many (and thus avoid the error)? This amounts to setting `max_categories=self.max_bins` in the internal `OrdinalEncoder`.

The workflow I would like to enable is fitting a HGB estimator when a categorical feature has a few too many categories, without needing to encode them myself beforehand.

### Describe your proposed solution

the HGB estimators would have a parameter (maybe something like `group_infrequent_categories`) to control whether they should display the current behavior (raise an error) or group together rare categories (with the OrdinalEncoder's `max_categories`) when there are more than `max_bins` categories

### Describe alternatives you've considered, if relevant

 It is relatively easy to apply an OrdinalEncoder before but (i) it is more verbose, and we have to use a pipeline and probably a ColumnTransformer, (ii) we lose the dtypes of pandas  dataframe columns so we have to specify the categorical columns manually instead of using `categorical_features="from_dtype"`, (iii) we end up doing the categorical feature encoding twice, once before and once inside the estimator

### Additional context

_No response_
user: jeromedockes
issue_27932: state: open
title: [WIP] Add Huber loss criterion to DecisionTreeRegressor and RandomForestRegressor
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #5368 (addresses the unfullifed request for Huber Loss)


#### What does this implement/fix? Explain your changes.
Adds Huber Loss as a valid `criterion` to the cited estimators.  Updated relevant unit tests to for the new `criterion`.


#### Any other comments?
Here is execution of the relevant CI tests after the modifications.
```
================================================== test session starts ===================================================
platform linux -- Python 3.10.12, pytest-7.4.0, pluggy-1.2.0
rootdir: /workspaces/scikit-learn
configfile: setup.cfg
plugins: cov-4.1.0
collected 1366 items                                                                                                     

../sklearn/tree/tests/test_export.py .............                                                                 [  0%]
../sklearn/tree/tests/test_monotonic_tree.py ..................................................................... [  6%]
.............................................................................                                      [ 11%]
../sklearn/tree/tests/test_reingold_tilford.py ..                                                                  [ 11%]
../sklearn/tree/tests/test_tree.py ............................................................................... [ 17%]
.................................................................................................................. [ 25%]
.................................................................................................................. [ 34%]
.................................................................................................................. [ 42%]
..........................................                                                                         [ 45%]
../sklearn/ensemble/tests/test_bagging.py ........................................................................ [ 50%]
..............................                                                                                     [ 53%]
../sklearn/ensemble/tests/test_base.py ..                                                                          [ 53%]
../sklearn/ensemble/tests/test_common.py ...................                                                       [ 54%]
../sklearn/ensemble/tests/test_forest.py ......................................................................... [ 60%]
.................................................................................................................. [ 68%]
.................................................................................................................. [ 76%]
............                                                                                                       [ 77%]
../sklearn/ensemble/tests/test_gradient_boosting.py ........................s..................................... [ 82%]
.................................................................                                                  [ 86%]
../sklearn/ensemble/tests/test_iforest.py .............................                                            [ 89%]
../sklearn/ensemble/tests/test_stacking.py ....................................................................... [ 94%]
                                                                                                                   [ 94%]
../sklearn/ensemble/tests/test_voting.py .............................                                             [ 96%]
../sklearn/ensemble/tests/test_weight_boosting.py ..................................................               [100%]

=============================== 1365 passed, 1 skipped, 212 warnings in 106.18s (0:01:46) ================================
vscode ➜ /workspaces/scikit-learn/sandbox (huber-loss) $ 
```
<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: jimthompson5802
issue_27931: state: open
title: ENH support for missing values in ExtraTrees 
body: ### Describe the workflow you want to enable

Inspired  by https://github.com/scikit-learn/scikit-learn/pull/26391 I think that support for missing values for ExtraTrees regressor and classifier should/could also be provided.

### Describe your proposed solution

I think a foundational work is already provided by @thomasjpfan  in https://github.com/scikit-learn/scikit-learn/pull/26391 and besides tests and documentation to enable nan handling it is enough to modify `sklearn/tree/_classes.py`:
For `ExtraTreeRegressor` add method:
```
 def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "squared_error",
            "friedman_mse",
            "poisson",
        }
        return {"allow_nan": allow_nan}
```
For `ExtraTreeClassifier` add method:
```
def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "gini",
            "log_loss",
            "entropy",
        }
        return {"multilabel": True, "allow_nan": allow_nan}
```
I've run the code locally, and it appears to be functioning as expected. However, I must emphasize that my testing was not exhaustive, and I might have overlooked some obvious aspects.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: mglowacki100
issue_27930: state: open
title: PR proposal to solve "Bunch object returns a regular dict when calling `copy` method on it"
body: ### Describe the bug

If I do

```python
bunch = Bunch (message='hello')
should_be_bunch = bunch.copy()
print (should_be_bunch.message)
```

I get a (for me) unexpected error, because `should_be_bunch` is actually a `dict`. This is easily fixable and I can submit a PR if this is of interest.

### Steps/Code to Reproduce

```python
from sklearn.utils import Bunch

bunch = Bunch (message='hello')
should_be_bunch = bunch.copy()
print (should_be_bunch.message)
```

### Expected Results

```pytb
Ouput in console: 
hello

No errors expected

### Actual Results

AttributeError                            Traceback (most recent call last)
Cell In[2], line 3
      1 bunch = Bunch (message='hello')
      2 should_be_bunch = bunch.copy()
----> 3 print (should_be_bunch.message)

AttributeError: 'dict' object has no attribute 'message'
```

### Versions

```shell
System:
    python: 3.10.9 (main, Mar  1 2023, 18:23:06) [GCC 11.2.0]
executable: /home/jaumeamllo/miniconda3/envs/tsforecast/bin/python
   machine: Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.2.1
          pip: 22.3.1
   setuptools: 65.6.3
        numpy: 1.23.5
        scipy: 1.10.0
       Cython: 0.29.33
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 2.2.0

Built with OpenMP: True
Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7f5e7a96cca0>
Traceback (most recent call last):
  File "/home/jaumeamllo/miniconda3/envs/tsforecast/lib/python3.10/site-packages/threadpoolctl.py", line 400, in match_module_callback
    self._make_module_from_path(filepath)
  File "/home/jaumeamllo/miniconda3/envs/tsforecast/lib/python3.10/site-packages/threadpoolctl.py", line 515, in _make_module_from_path
    module = module_class(filepath, prefix, user_api, internal_api)
  File "/home/jaumeamllo/miniconda3/envs/tsforecast/lib/python3.10/site-packages/threadpoolctl.py", line 606, in __init__
    self.version = self.get_version()
  File "/home/jaumeamllo/miniconda3/envs/tsforecast/lib/python3.10/site-packages/threadpoolctl.py", line 646, in get_version
    config = get_config().split()
AttributeError: 'NoneType' object has no attribute 'split'

threadpoolctl info:
       filepath: /home/jaumeamllo/miniconda3/envs/tsforecast/lib/libmkl_rt.so.1
         prefix: libmkl_rt
       user_api: blas
   internal_api: mkl
        version: 2021.4-Product
    num_threads: 2
threading_layer: intel

       filepath: /home/jaumeamllo/miniconda3/envs/tsforecast/lib/libgomp.so.1.0.0
         prefix: libgomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 4

       filepath: /home/jaumeamllo/miniconda3/envs/tsforecast/lib/libiomp5.so
         prefix: libiomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 4
```

user: JaumeAmoresDS
issue_27928: state: open
title: LASSO Solve badly when alpha is extremely small
body: ### Describe the bug

There are 2 problem:

- when `tol=1e-4`(default), the solver does not give a warning when it solved badly.
- when `alpha` is extrimely small (like 1e-8), the solver could not find solution properly.

### Steps/Code to Reproduce

In this case, solver even do not raise a warning until `tol` = 1e-5.
```python
from sklearn.linear_model import MultiTaskLasso
import numpy as np

seed = 114514
np.random.seed(seed)

m = 256
n = 512
l = 2
k = int(m*0.1)

X = np.random.randn(m, n)
u = np.zeros((n, l))
# randomly choose k rows of u
u[np.random.choice(n, k, replace=False), :] = np.random.randn(k, l)
y = np.dot(X, u)

def loss(w:np.ndarray):
    return 0.5/m * np.linalg.norm(y - np.dot(X, w), ord='fro')**2 + 0.01 * np.linalg.norm(w, ord=1, axis=0).sum()

def err_to_exact(w:np.ndarray):
    return np.linalg.norm(w - u, ord='fro')

def err_fn(w1:np.ndarray,w2:np.ndarray):
    return np.linalg.norm(w1 - w2, ord='fro')

clf = MultiTaskLasso(alpha=0.01/m)
# clf = MultiTaskLasso(alpha=0.01/m,tol=1e-5)
clf.fit(X, y)
x = clf.coef_.T
print("dual gap:",clf.dual_gap_)
print("origin u loss:",loss(u))
print("iter:{}, loss:{}, err-to-exact:{}".format(clf.n_iter_,loss(x),err_to_exact(x)))
```


### Expected Results

To solve the problem when `tol` is quite small, we should do warm up training and train the model from larger $\alpha$ to smaller.
temp solution:
```python
clf = MultiTaskLasso(alpha=1/m,tol=1e-5,warm_start=True)
for i in range(3):
    clf.fit(X,y)
    clf.alpha /= 10
```
output:
```
dual gap: 6.679412639977893e-05
origin u loss: 0.5091769015638326
iter:6, loss:0.5091640128686069, err-to-exact:0.00021114026824291122
```
should we add it into .fit method?

### Actual Results

when $tol=1e-4,\alpha=0.01/m$:
```
dual gap: 0.0060915764336923126
origin u loss: 0.5091769015638326
iter:20, loss:2.2204027219101863, err-to-exact:9.758212052573763
```
The loss is quite larger than the origin u, so it must be wrong.
when $tol=1e-5,\alpha=0.01/m$:
```
Warning: Objective did not converge. You might want to increase the number of iterations. 
Duality gap: 1.3354989080418382, tolerance: 0.22307386842917606
  ) = cd_fast.enet_coordinate_descent_multi_task(
dual gap: 0.005216792609538431
origin u loss: 0.5091769015638326
iter:1000, loss:1.9859143908713657, err-to-exact:9.067098629992879
```

### Versions

```shell
1.2.2
```

user: Rain1560
issue_27927: state: open
title: `classification_report` gives micro averages when `labels` is a superset of the observed labels 
body: ### Describe the bug

When the value of the `labels` parameter is a superset of all observed classes in `y_true` and `y_pred`, `classification_report()` gives separate macro average values for precision, recall, and F1, although according to [the documentation](https://scikit-learn.org/1.3/modules/generated/sklearn.metrics.classification_report.html) this should only be done when `labels` is a subset of the observed classes.

### Steps/Code to Reproduce

```python
from sklearn.metrics import classification_report
print(classification_report([0, 1], [1, 0], labels=[0, 1, 2], zero_division=0.0))
```

### Expected Results

```
              precision    recall  f1-score   support

           0       0.00      0.00      0.00       1.0
           1       0.00      0.00      0.00       1.0
           2       0.00      0.00      0.00       0.0

    accuracy                           0.00       2.0
   macro avg       0.00      0.00      0.00       2.0
weighted avg       0.00      0.00      0.00       2.0
```

### Actual Results

```
              precision    recall  f1-score   support

           0       0.00      0.00      0.00       1.0
           1       0.00      0.00      0.00       1.0
           2       0.00      0.00      0.00       0.0

   micro avg       0.00      0.00      0.00       2.0
   macro avg       0.00      0.00      0.00       2.0
weighted avg       0.00      0.00      0.00       2.0
```

### Versions

```shell
System:
    python: 3.11.5 (main, Sep 11 2023, 13:54:46) [GCC 11.2.0]
executable: /home/connor/miniconda3/envs/playground/bin/python
   machine: Linux-6.2.0-37-generic-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.3.2
          pip: 23.3
   setuptools: 68.0.0
        numpy: 1.26.2
        scipy: 1.11.4
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: libgomp
       filepath: /home/connor/miniconda3/envs/playground/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libopenblas
       filepath: /home/connor/miniconda3/envs/playground/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Zen

       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libopenblas
       filepath: /home/connor/miniconda3/envs/playground/lib/python3.11/site-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Zen
```

user: boyleconnor
issue_27913: state: open
title: Added link to plot_adaboost_multiclass.py example
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs

Towards #26927 
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.

Includes a link to the <code>plot_adaboost_multiclass.py</code> example


#### Any other comments?

NA


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: virchan
issue_27911: state: open
title: Attempt to speed up unique value discovery in `_BaseEncoder` for polars and pandas series
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Address this [comment](https://github.com/scikit-learn/scikit-learn/pull/27835#pullrequestreview-1759927546) in a follow-up PR to #27835 

#### What does this implement/fix? Explain your changes.

this relies on the pandas or polars Series `unique` method rather than `numpy.unique` to identify categories as they can be faster.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

so far I am seeing a speedup for pandas but not really for polars; unless I can make it faster for polars it is probably not worth the added complexity

<details>
<summary>toy benchmark</summary>

https://gist.github.com/jeromedockes/d2e1fc147b7ad0a6dfd686318cc9da57

results

<pre>
branch: main                            branch: ordinal-encoder-pd-unique

polars                                  polars
========                                ========
ordinal encoder fit: 7.05e-02           ordinal encoder fit: 4.89e-02
gradient boosting fit: 3.74e-01         gradient boosting fit: 3.19e-01
_unique(array): 1.38e-02                _unique(series): 1.92e-04
series.unique(): 2.55e-05               series.unique(): 2.56e-05
np.unique(): 4.84e-01                   np.unique(): 4.69e-01

pandas                                  pandas
========                                ========
ordinal encoder fit: 1.40e-02           ordinal encoder fit: 5.72e-03
gradient boosting fit: 2.90e-01         gradient boosting fit: 2.55e-01
_unique(array): 1.03e-02                _unique(series): 2.39e-03
series.unique(): 2.36e-03               series.unique(): 2.38e-03
np.unique(): 3.25e-01                   np.unique(): 3.25e-01

</pre>

if we change the type of the categorical column to contain integers rather than categories we see a small speedup for polars but almost 10x for the OrdinalEncoder on pandas

<pre>
branch: main                            branch: ordinal-encoder-pd-unique

polars                                  polars
========                                ========
ordinal encoder fit: 2.46e-02           ordinal encoder fit: 1.23e-02
gradient boosting fit: 2.42e-01         gradient boosting fit: 2.67e-01
_unique(array): 2.09e-02                _unique(series): 1.11e-02
series.unique(): 8.90e-03               series.unique(): 8.44e-03
np.unique(): 2.08e-02                   np.unique(): 2.07e-02

pandas                                  pandas
========                                ========
ordinal encoder fit: 2.11e-02           ordinal encoder fit: 2.89e-03
gradient boosting fit: 2.49e-01         gradient boosting fit: 2.32e-01
_unique(array): 2.09e-02                _unique(series): 2.70e-03
series.unique(): 2.69e-03               series.unique(): 2.69e-03
np.unique(): 2.08e-02                   np.unique(): 2.07e-02
</pre>

</details>

user: jeromedockes
issue_27908: state: open
title: Giving credits to Rosenblatt, the creator of perceptron in doc/modules/linear_model.rst
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### What does this implement/fix? Explain your changes.
I have made a small change in the documentation of perceptron, linking the paper of Rosenblatt, the creator.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: AlessandroTesi8
issue_27905: state: open
title: Ensure predictions sparse before `sp.hstack` in `ClassifierChain`
body: We use `sp.hstack` in a number of places in `ClassifierChain` where we may be stacking sparse with dense, e.g.,:

https://github.com/scikit-learn/scikit-learn/blob/36f6734789fc7e4940792c1cfb6a6e90dfcae484/sklearn/multioutput.py#L948

and

https://github.com/scikit-learn/scikit-learn/blob/36f6734789fc7e4940792c1cfb6a6e90dfcae484/sklearn/multioutput.py#L693

AFAICT it seems stacking a sparse with dense via `sp.hstack` gives you a sparse array (even though `sp.hstack` is not documented to support dense):

```bash
In [34]: from scipy.sparse import coo_matrix, hstack
    ...: 
    ...: A = coo_matrix([[1, 2], [3, 4]])

In [35]: B = np.zeros((2,2))

In [36]: hstack([A,B])
Out[36]: 
<2x4 sparse matrix of type '<class 'numpy.float64'>'
        with 4 stored elements in COOrdinate format>
```

Maybe due to: https://github.com/scipy/scipy/blob/f990b1d2471748c79bc4260baf8923db0a5248af/scipy/sparse/_construct.py#L654 ?

Should we ensure y is sparse before using `sp.hstack` ?

I had quick look at our code, I could not find any other cases where it would be possible to be stacking dense + sparse. I think `ClassifierChain` is unique in that we do not usually combine `X` with `y`

Discussed here: https://github.com/scikit-learn/scikit-learn/pull/27700#discussion_r1378691272

cc @glemaitre 
user: lucyleeow
labels: Needs Triage: Issue requires triage
issue_27904: state: open
title: ENH Use Array API in `r2_score`
body: #### Reference Issues/PRs

The PR builds on preliminary explorations done by @elindgren in https://github.com/scikit-learn/scikit-learn/pull/27102

It tackles one of the items outlined in https://github.com/scikit-learn/scikit-learn/issues/26024.

#### Any other comments?

This PR proposes to fallbacks to cpu+numpy at the very beginning of the r2_score function whenever the array namespace and the device can't handle float64 precision, because explicit castings to float64 are unavoidable and are used in a lot of steps.

It also proposes improved ways to detect device support for dtypes, and uses it to act accordingly in r2_score and _average, but also updates weighted_sum function.

user: fcharras
labels: Waiting for Second Reviewer: First reviewer is done, need a second one!
issue_27903: state: open
title: allow_nan tag in Pipelines
body: Unfortunately, our tag system for allowing nans does not work with pipelines. Lets say we have a pipeline with two steps and the final step does not accept nans:

1. If the first step is an Imputer, then the pipeline accept nans. For example: `make_pipeline(SimpleImputer(), LogisticRegression())`
2. If the first step is a StandardScalar (which accept nans and leaves them along), then the pipeline does not accept nans. For example: `make_pipeline(StandardScalar(), LogisticRegression())`

We likely need a "output_nan" tag to reliability give a pipeline a "allow_nan" tag.
user: thomasjpfan
labels: Needs Decision - API: Requires decision regarding API
issue_27891: state: open
title: Add inverse-transform to _set_output
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes #27843
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
The problem is mentioned in the issue linked above. Here is how the current solution works.
```python
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_breast_cancer

X, _ = load_breast_cancer(return_X_y=True, as_frame=True)

scaler = StandardScaler().set_output(transform="pandas").fit(X)
Xt = scaler.transform(X)

print(scaler.inverse_transform(Xt))
```
The above code will print a DataFrame.

<!--
#### Any other comments?
-->

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: sky-2002
issue_27890: state: open
title: Adding the median option to the KNN imputer.
body: Current version only supports mean but  there seems to be a median version showing good results. So , I added it to the repo using the "strategy" argument

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.

Imputer could only have a mean option in the earlier version.

#### Any other comments?
Apologies if I made some error in the pull request, I am still learning.
Please let me know if there is anything I missed or needs to be changed.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: blueprintparadise
issue_27882: state: open
title: [RFC] Varying the number of outputs considered for splitting in Multi Output Decision Trees
body: ### Describe the workflow you want to enable

One strength of RFRs is that they are incredibly robust and therefore provide a strong baseline for many tasks without needing to consider normalization or scaling of either the inputs or outputs. In the case of multi-output RFRs this robustness towards the output space goes away due to the summing of impurities across different output dimensions which entails the need to standardize the output labels to ensure that undue attention isn't given to particular outputs. Currently the documentation doesn't readily inform the user of this artifact. In the spirit of the Random Forest one solution to avoiding this problem would be to randomly sample which output(s) to consider for the determination of the split. If the number of outputs was set to 1 then we would end up with a case where the normalization of the output space once again doesn't matter.

### Describe your proposed solution

The introduction of a new kwarg `max_outputs` (by analogy to `max_features`) could allow users to control how many outputs were considered when selecting the optimal split. If set to `1.0` all outputs would be used as currently, if set to `1` then a single output would be used as described above. This seems like a relevant and natural hyper-parameter for the multi-output RFR.  

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I have not been able to find literature that explores the above slight adjustment to the current algorithm. This is a RFC to see if the team would accept such a PR in principle without the quoted 200+ citation requirement if sufficient empirical evidence was provided. 
user: CompRhys
labels: Needs Reproducible Code: Issue requires reproducible code
issue_27881: state: open
title: [RFC] Leaf Level Variance in Multi Output Decision Trees
body: ### Describe the workflow you want to enable

For single output RFR trained with the squared error criterion the impurity of the leaves can be used as a crude but useful estimate of the aleatoric uncertainty. In the multi output case the impurity is the sum over outputs hence this is no longer possible to estimate.

### Describe your proposed solution

Currently we can access the node values from `estimator.tree_.values` that stores the leaf means, I propose adding another data store or `estimator.tree_.square_means` that would store the square means of the leaves. Using these it is possible to work out the variance within the leaf that can then be used as a crude but useful estimate of the aleatoric uncertainty. The added benefit of doing this would be that this could be calculated for any choice of criterion rather than just the squared error. The principal issue is that it would double the amount of data stored.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This measure of the aleatoric uncertainty is widely used in the work of frank hutter in more recent versions of SMAC - https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/11-LION5-SMAC.pdf
user: CompRhys
labels: Needs Decision: Requires decision
issue_27880: state: open
title: DOC replace MAPE in lagged features example
body: A few improvements could be made on the new example of #25350:
- Mean absolute percentage error (MAPE) is used quite a lot. I propose to replace it, in particular if predicting/forecasting the mean value. Note that MAPE is optimized by the median of a distribution with pdf propotional to $\frac{f(y)}{y}$, where $f(y)$ is the pdf of the true distribution of the data.

- The `pinball_loss_50` is the same as `1/2 MAE`, this redundancy could be removed.

- A residual vs predicted does note really make sense for 5%- and 95%-quantile prediction.
  A reliability diagram for quantiles might be a  good replacement, see [model-diagnostics plot_reliability_diagram](https://lorentzenchr.github.io/model-diagnostics/reference/model_diagnostics/calibration/plots/#model_diagnostics.calibration.plots.plot_reliability_diagram). Note that this is not possible within current scikit-learn. Maybe the best next action is to add a little more explanation to the graphs.
user: lorentzenchr
issue_27879: state: open
title: Pandas Copy-on-Write mode should be enabled in all tests
body: ### Describe the bug

Pandas COW will be enabled by default in version 3.0.
For example, today I just found that `TargetEncoder` doesn't work properly with it enabled.
There are probably many other examples that could be uncovered by testing.

### Steps/Code to Reproduce

```python
import pandas as pd
from sklearn.preprocessing import TargetEncoder
pd.options.mode.copy_on_write = True

df = pd.DataFrame({
    "x": ["a", "b", "c", "c"],
    "y": [4., 5., 6., 7.]
})
t = TargetEncoder(target_type="continuous")
t.fit(df[["x"]], df["y"])
```

### Expected Results

No error.

### Actual Results
```
ValueError                                Traceback (most recent call last)
Cell In[2], line 10
      5 df = pd.DataFrame({
      6     "x": ["a", "b", "c", "c"],
      7     "y": [4., 5., 6., 7.]
      8 })
      9 t = TargetEncoder(target_type="continuous")
---> 10 t.fit(df[["x"]], df["y"])

File ~/.conda/envs/jhop311/lib/python3.11/site-packages/sklearn/base.py:1152, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1145     estimator._validate_params()
   1147 with config_context(
   1148     skip_parameter_validation=(
   1149         prefer_skip_nested_validation or global_skip_validation
   1150     )
   1151 ):
-> 1152     return fit_method(estimator, *args, **kwargs)

File ~/.conda/envs/jhop311/lib/python3.11/site-packages/sklearn/preprocessing/_target_encoder.py:203, in TargetEncoder.fit(self, X, y)
    186 @_fit_context(prefer_skip_nested_validation=True)
    187 def fit(self, X, y):
    188     """Fit the :class:`TargetEncoder` to X and y.
    189 
    190     Parameters
   (...)
    201         Fitted encoder.
    202     """
--> 203     self._fit_encodings_all(X, y)
    204     return self

File ~/.conda/envs/jhop311/lib/python3.11/site-packages/sklearn/preprocessing/_target_encoder.py:332, in TargetEncoder._fit_encodings_all(self, X, y)
    330 if self.smooth == "auto":
    331     y_variance = np.var(y)
--> 332     self.encodings_ = _fit_encoding_fast_auto_smooth(
    333         X_ordinal, y, n_categories, self.target_mean_, y_variance
    334     )
    335 else:
    336     self.encodings_ = _fit_encoding_fast(
    337         X_ordinal, y, n_categories, self.smooth, self.target_mean_
    338     )

File sklearn/preprocessing/_target_encoder_fast.pyx:82, in sklearn.preprocessing._target_encoder_fast._fit_encoding_fast_auto_smooth()

File stringsource:660, in View.MemoryView.memoryview_cwrapper()

File stringsource:350, in View.MemoryView.memoryview.__cinit__()

ValueError: buffer source array is read-only
```

### Versions

```shell
System:
    python: 3.11.3 | packaged by conda-forge | (main, Apr  6 2023, 08:57:19) [GCC 11.3.0]
executable: /home/jhopfens/.conda/envs/jhop311/bin/python
   machine: Linux-3.10.0-1160.99.1.el7.x86_64-x86_64-with-glibc2.17

Python dependencies:
      sklearn: 1.3.2
          pip: 23.0.1
   setuptools: 67.6.1
        numpy: 1.25.2
        scipy: 1.11.2
       Cython: 3.0.0
       pandas: 2.1.0
   matplotlib: 3.7.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/jhopfens/.conda/envs/jhop311/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-5007b62f.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 64

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/jhopfens/.conda/envs/jhop311/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 128

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/jhopfens/.conda/envs/jhop311/lib/python3.11/site-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 64
```

user: s-banach
issue_27878: state: open
title: [WIP] Implement Ridge CCA in Cross Decomposition Module
body: #### What does this implement do? Explain your changes.

This pull request implements the Ridge CCA (RidgeCCA) algorithm in the sklearn.cross_decomposition module. Ridge CCA slides between an unregularized (CCA) solution and a maximally regularized (PLS) solution. This implementation includes:

- The Ridge CCA algorithm with options to set the number of components and ridge parameters for x and y (we use \alpha to match with Ridge Regression in scikit-learn).

- Appropriate unit tests to ensure functionality and compliance with sklearn standards.

- Documentation updates, including a description of Ridge CCA, its use cases, and a comparison with other CCA/PLS methods.

Any other comments?

I expect this to be really useful in lots of practical applications. Using PLS or CCA is analagous to only having access to LinearRegression or Ridge with \alpha=1,000,000,000 i.e. a pure covariance based regression rather than correlation. In fact CCA with a univariate y is equivalent in a sense to LinearRegression.

For this reason, the performance of a tuned Ridge CCA is obviously lower bounded by PLS and CCA but in practice will almost always outperform them both (analagous with ridge regression).

A subtle implementation detail choice is whether to use (CCA) 0<alpha_x,alpha_y<1 (PLS) , or (CCA) 0<alpha_x,alpha_y<unbounded (PLS in the limit as alpha-> infinity). Both have been used in the literature. The latter is closer to the sklearn ridge regression, while the former better highlights the sliding between CCA and PLS.

Thanks for reviewing this contribution!

[1] Mihalik, A., Chapman, J., Adams, R. A., Winter, N. R., Ferreira, F. S., Shawe-Taylor, J., ... & Alzheimer’s Disease Neuroimaging Initiative. (2022). Canonical correlation analysis and partial least squares for identifying brain-behaviour associations: a tutorial and a comparative study. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging.
user: jameschapman19
issue_27875: state: open
title: ENH Use scipy.special.inv_boxcox in PowerTransformer
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.

Since we already use `scipy.special.boxcox`, I think we can use `scipy.special.inv_boxcox` for the inverse too.
https://github.com/scipy/scipy/blob/fcf7b652bc27e47d215557bda61c84d19adc3aae/scipy/special/_boxcox.pxd#L30-L34

#### Any other comments?

The Box-Cox transformation.
![image](https://github.com/scikit-learn/scikit-learn/assets/41134380/8f280d65-77ef-4860-8b29-0da30cf9755a)


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: xuefeng-xu
issue_27873: state: open
title: RFC Unify old GradientBoosting estimators and HGBT
body: ### Current situation
We have the unfortunate situation to have 2 different versions of gradient boosting, the old estimators ([`GradientBoostingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn-ensemble-gradientboostingclassifier) and `GradientBoostingRegressor`) as well as the new ones using binning and histogram strategies similar to LightGBM ([`HistGradientBoostingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingClassifier.html#sklearn-ensemble-histgradientboostingclassifier) and `HistGradientBoostingRegressor`).

This makes advertising the new ones harder, e.g. #26826, and also result in a larger feature gap between those two.
Based on discussions in #27139 and during a monthly meeting (maybe not documented), **I'd like to call for comments on the following:**

#### Proposition
Unify both types of gradient boosting in a single class, i.e. the old names `GradientBoostingClassifier` and make them switch the underlying estimator class based on a parameter value, e.g. `max_bins` (`None`->old classes, integer->new classes).

Note that binning and histograms are not the only difference.

### Comparison
#### Algorithm
The old GBT uses Friedman gradient boosting with a line search step. (The lines search sometimes, e.g. for log loss, uses a 2. order approximation and is therefore, sometimes, called "hybrid gradient-Newton boosting"). The trees are learned on the gradients. A tree searches for the best split among all (veeeery many) split candidates for all features. After a single tree is fit, the terminal node values are re-computed which corresponds to a line search step.

The new HGBT uses a 2. order approximation of the loss, i.e. gradients and hessians (XGBoost paper, therefore sometimes called Newton boosting). In addition, it bins/discretizes the features `X` and uses a histogram of gradients/hessians/counts per feature. A tree then searches for the best split candidate, but there are only n_features * n_bins candidates (muuuuch less than in GBT).

| estimator | trees train on        | node values (consequence of tree train) | features `X`   |
| --------- | --------------------- | ---------------------------- | -------------- |
| GBT       | gradients             | recomputed in lines search   | use as is      |
| HGBT      | gradients/hessians    | `sum(gradient)/sum(hessian)` | bin/discretize |

In fact, one could use 2. order loss (gradients and hessians) without binning `X`, and vice-versa, use binning with fitting trees on gradients (without hessians).

#### Parameters

| `HistGradientBoostingRegressor` | `GradientBoostingRegressor` | Same               | Comment |
| ------------------------------- | ----------------------------| ------------------ | ------- |
| loss                            | loss                        | :white_check_mark: | |
| quantile                        | alpha                       | :x:                | | 
| learning_rate                   | learning_rate               | :white_check_mark: | |
| max_iter                        | n_estimators                | :x:                | https://github.com/scikit-learn/scikit-learn/pull/12807#issue-391902947 |
| max_leaf_nodes                  | max_leaf_nodes              | :white_check_mark: |
| max_depth                       | max_depth                   | :white_check_mark: |
| min_samples_leaf                | min_samples_leaf            | :white_check_mark: |
| l2_regularization               | learning_rate               | :white_check_mark: |
| max_features                    | max_features                | :white_check_mark: |
| max_bins                        | :no_entry: (nonsense)       | :x:                |
| categorical_features            | :no_entry:                  | :x:                |
| monotonic_cst                   | :no_entry:                  | :x:                | #27305 |
| interaction_cst                 | :no_entry:                  | :x:                |
| warm_start                      | warm_start                  | :white_check_mark: |
| early_stopping                  | :no_entry:                  | :x:                |
| scoring                         | :no_entry:                  | :x:                |
| validation_fraction             | validation_fraction         | :white_check_mark: |
| n_iter_no_change                | n_iter_no_change            | :white_check_mark: |
| tol                             | tol                         | :white_check_mark: |
| verbose                         | verbose                     | :white_check_mark: |
| random_state                    | random_state                | :white_check_mark: |
| class_weight                    | :no_entry:                  | :x:                |
| :no_entry:                      | subsample                   | :x:                | #16062 |
| :no_entry: (nonsense)           | criterion                   | :x:                |
| :no_entry:                      | min_samples_split           | :x:                |
| :no_entry:                      | min_weight_fraction_leaf    | :x:                |
| :no_entry:                      | min_impurity_decrease       | :x:                |
| :no_entry:                      | init                        | :x:                | #27109 |
| :no_entry:                      | ccp_alpha                   | :x:                |

In fact, only the quantile/alpha and max_iter/n_estimator parameters are conflicting.

user: lorentzenchr
issue_27868: state: open
title: ENH use log1p and expm1 in Yeo-Johnson transformation and its inverse
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.

This PR was inspired by scipy's YJ transformation and also implement its inverse.
https://github.com/scipy/scipy/blob/fcf7b652bc27e47d215557bda61c84d19adc3aae/scipy/stats/_morestats.py#L1495-L1516

Specifically, if $\lambda=1$, we could skip the computation and return x directly.

#### Any other comments?
The formula of YJ transformation
![image](https://github.com/scikit-learn/scikit-learn/assets/41134380/0d2abc91-d8f6-4bd3-9fd7-e9d12c8209a7)



<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: xuefeng-xu
issue_27866: state: open
title: FIX handle single 1D input to sample_y before fit is called
body: #### Reference Issues/PRs
none


#### What does this implement/fix? Explain your changes.
There is an issue when calling `y_sample` with a single 1D sample before `fit` is called with training data.  The `y_mean` returned by the call to `predict` returns an invalid size when the `squeeze` method removes all dimensions <=1.  

#### Any other comments?

I have not extensively tested and maybe a conditional branch like that found on line 448 is better?


user: constant5
issue_27846: state: open
title: ⚠️ CI failed on Ubuntu_Atlas.ubuntu_atlas ⚠️
body: **CI is still failing on [Ubuntu_Atlas.ubuntu_atlas](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=63259&view=logs&j=689a1c8f-ff4e-5689-1a1a-6fa551ae9eba)** (Jan 31, 2024)
- test_pca_sparse_fit_transform[21-csr_array]
- test_pca_sparse_fit_transform[21-csc_array]
user: scikit-learn-bot
issue_27843: state: open
title: set_output doesn't work for inverse_transform method
body: ### Describe the bug

Using `set_output(transfrom="pandas")` doesn't return a pandas dataframe for the StandardScaler's `inverse_transform` method.

### Steps/Code to Reproduce

```python
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_breast_cancer

X, _ = load_breast_cancer(return_X_y=True, as_frame=True)

scaler = StandardScaler().fit(X)
Xt = scaler.transform(X)

print(scaler.inverse_transform(Xt))
```

### Expected Results

I expect a pd.DataFrame in return, just like with the `transform` method.

### Actual Results

A numpy array.

```
[[1.799e+01 1.038e+01 1.228e+02 ... 2.654e-01 4.601e-01 1.189e-01]
 [2.057e+01 1.777e+01 1.329e+02 ... 1.860e-01 2.750e-01 8.902e-02]
 [1.969e+01 2.125e+01 1.300e+02 ... 2.430e-01 3.613e-01 8.758e-02]
 ...
 [1.660e+01 2.808e+01 1.083e+02 ... 1.418e-01 2.218e-01 7.820e-02]
 [2.060e+01 2.933e+01 1.401e+02 ... 2.650e-01 4.087e-01 1.240e-01]
 [7.760e+00 2.454e+01 4.792e+01 ... 0.000e+00 2.871e-01 7.039e-02]]
```

### Versions

```shell
System:
    python: 3.11.2 (tags/v3.11.2:878ead1, Feb  7 2023, 16:38:35) [MSC v.1934 64 bit (AMD64)]
executable: C:\Users\Mavs\Documents\Python\ATOM\venv311\Scripts\python.exe
   machine: Windows-10-10.0.19045-SP0
Python dependencies:
      sklearn: 1.3.2
          pip: 23.3.1
   setuptools: 68.2.2
        numpy: 1.24.4
        scipy: 1.11.3
       Cython: 3.0.5
       pandas: 2.1.2
   matplotlib: 3.8.0
       joblib: 1.3.2
threadpoolctl: 3.2.0
Built with OpenMP: True
threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: C:\Users\Mavs\Documents\Python\ATOM\venv311\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
        version: 0.3.21
threading_layer: pthreads
   architecture: Zen
       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: vcomp
       filepath: C:\Users\Mavs\Documents\Python\ATOM\venv311\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: C:\Users\Mavs\Documents\Python\ATOM\venv311\Lib\site-packages\scipy.libs\libopenblas_v0.3.20-571-g3dec11c6-gcc_10_3_0-c2315440d6b6cef5037bad648efc8c59.dll
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Zen
       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libiomp
       filepath: C:\Users\Mavs\Documents\Python\ATOM\venv311\Lib\site-packages\torch\lib\libiomp5md.dll
        version: None
       user_api: openmp
   internal_api: openmp
    num_threads: 1
         prefix: libiomp
       filepath: C:\Users\Mavs\Documents\Python\ATOM\venv311\Lib\site-packages\torch\lib\libiompstubs5md.dll
        version: None
```

user: tvdboom
issue_27842: state: open
title: [MRG] Polynomial Chaos Expansions
body: *Note: I've labeled this PR as [MRG], following [this section](https://scikit-learn.org/stable/developers/contributing.html#pull-request-checklist) from the Developer's Guide, but please let me know if this is inappropriate at this stage.*

## What is it?

The [Polynomial Chaos expansion (PCE)](https://en.wikipedia.org/wiki/Polynomial_chaos) is a supervised learning technique that is used to represent and propagate uncertainties in complex models. It is a well-known tool for sensitivity analysis and uncertainty quantification, that has been used in a variety of applications, see, for example, [[Ghanem1991](https://books.google.com/books/about/Stochastic_Finite_Elements.html?id=WzgKyTQQcAwC)] and [[Najm2009](https://www.annualreviews.org/doi/abs/10.1146/annurev.fluid.010908.165248)].

The basic idea is to represent the parameters (input features) of the model as random variables with an associated probability density function. The model output, which is also a random variable, is then expressed as a polynomial function of the input random variables.

The Polynomial Chaos methodology has found widespread use in engineering and applied sciences in the last decades, because it presents a systematic and efficient framework for dealing with parameteric uncertainty in mathematical models.

This is a proposal for an implementation that adds Polynomial Chaos expansions to `scikit-learn`.

## How does it work?

Given a set of input features $X$, sampled from an assumed distribution, the Polynomial Chaos approximation for the model output $y = f(X)$ is

$$ y = \sum_{j} c_j \Psi_j(X),$$

where $\Psi_j$ is a multivariate orthogonal polynomial basis function, and $c_j$ is a coefficient in the expansion. The type of orthogonal polynomial depends on the assumed distribution for the input features. For example, the [uniform distribution](https://en.wikipedia.org/wiki/Continuous_uniform_distribution) is associated with the [Legendre](https://en.wikipedia.org/wiki/Legendre_polynomials) polynomials, the [normal distribution](https://en.wikipedia.org/wiki/Normal_distribution) is associated with the [Hermite](https://en.wikipedia.org/wiki/Hermite_polynomials) polynomials, and other distributions can be matched to similar orthogonal polynomials from the [Askey scheme](https://en.wikipedia.org/wiki/Askey_scheme), see [[Xiu2002](https://epubs.siam.org/doi/pdf/10.1137/S1064827501387826)].

![polynomials](https://github.com/scikit-learn/scikit-learn/assets/15999604/a1cf5cb4-8a24-4f2d-9b6c-8f1952a8442f)

*(Note that I'm deliberately ignoring some mathematical details in this description - I do take care of these in the actual implementation.)*

## Implementation details

In `scikit-learn` language, the PCE can be interpreted as a pipeline with a nonlinear [tranformer](https://scikit-learn.org/stable/data_transforms.html) that transforms the input features into orthogonal polynomial features, followed by a [linear regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) step:

```python
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OrthogonalPolynomialFeatures # new

pce = Pipeline([
    ('transformer', OrthogonalPolynomialFeatures()),
    ('linear_model', LinearRegression())
])
```

Thus, this PR introduces two new classes: `OrthogonalPolynomialFeatures` and `PolynomialChaosRegressor`. The former is a transformer class that is an extension of the existing [PolynomialFeatures](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html) class, but uses orthogonal polynomials instead of simple monomials. The latter is a [`BaseEstimator`](https://scikit-learn.org/stable/modules/generated/sklearn.base.BaseEstimator.html) that implements the pipeline described above.

After fitting, the PCE can be used as a surrogate model to replace further evaluations of the model problem (using the [`predict`](https://scikit-learn.org/stable/tutorial/statistical_inference/supervised_learning.html) method from the `BaseEstimator` class).

## Why add Polynomial Chaos expansions to `scikit-learn`?

Existing tools for constructing PCEs are either not written in Python ([uqlab](https://www.uqlab.com/), [dakota](https://dakota.sandia.gov/about-dakota/)), have poor performance in high dimensions ([chaospy](https://github.com/jonathf/chaospy)) or have complex API designs ([pygpc](https://github.com/pygpc-polynomial-chaos/pygpc)). Recognizing these limitations, I believe there is a clear opportunity for a well-designed PCE tool that overcomes these issues.

The construction scheme (a pipeline with a nonlinear `transformer` followed by a `linear_model`) outlined in the previous section, aligns seamlessly with `scikit-learn`s modular API. In addition, we can use the many [`linear_model`](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.linear_model)s in `scikit-learn` to help avoid overfitting, a common problem when constructing PCEs with noisy and/or limited data. It's worth mentioning that `chaospy` and `pygpc` already rely on `scikit-learn` estimators for similar purposes.

To illustrate the efficiency statement above, we compare the performance of `chaospy`, `pygpc` and the [scikit-learn implementation](https://github.com/PieterjanRobbe/scikit-learn/tree/polynomial_chaos) by constructing a PCE for the Sobol function in 4 dimensions (i.e., 4 input features). The [Sobol function](https://www.sfu.ca/~ssurjano/gfunc.html) is a well-known test function in sensitivity analysis and uncertainty quantification.

![timing-dim4](https://github.com/scikit-learn/scikit-learn/assets/15999604/47588ae3-bd65-4615-9020-e98d8cb15e9b)

The figure on the left shows the time needed to construct the PCE as a function of the maximum degree of the polynomials in the expansion. The figure on the right shows the same, but now with a logarithmic vertical axis. The `scikit-learn` implementation is significantly faster than the established tools, depite the reasonably moderate dimension. When considering dimensions other than 4, we observe a similar or even better speedup, see [this notebook](https://github.com/PieterjanRobbe/scikit-learn-polynomial-chaos-benchmarks/blob/main/comparison.ipynb).

## Global Sensitivity Analysis

A common postprocessing step after the Polynomial Chaos expansion has been fitted is to perform a [Global Sensitivity Analysis](https://en.wikipedia.org/wiki/Sensitivity_analysis) by computing the so-called [Sobol sensitivity indices](https://en.wikipedia.org/wiki/Variance-based_sensitivity_analysis). These indices rank the input features by importance, by computing what fraction of the variance of the output they explain. When a PCE surrogate model has been fitted, these indices can be computed analytically, see for example [[Sudret2008](https://www.sciencedirect.com/science/article/pii/S0951832007001329)].

In the proposed implementation, the sensitivity indices can be computed by calling the `main_sens()` and `total_sens()` methods against an instance of the `PolynomialChaosRegressor` class.

As a side note, I found [this PR](https://github.com/scikit-learn/scikit-learn/issues/22453) that also discusses Global Sensitivity Analysis. However, the method proposed in the latter PR is a pure sampling-based method, and sensitivities extracted from Polynomial Chaos expansions can be a lot more reliable (see the Ishigami example in the documentation for a comparison between the two).

## Motiviation

This porposal has mainly been written because of my disappointment with the existing tools. I regularly use Polynomial Chaos expansions for research purposes, and not having a reliable, scalable and user-friendly implementation available has hurt me on more than one occasion. Hence the motivation to supply my own implementation.

I sincerely welcome feedback and collaborations to make this porposal a valuable addition to the `scikit-learn` ecosystem. Thanks!

## References

The [**number**] after each reference indicates the number of citations (following Google scholar).

[[Ghanem1991](https://books.google.com/books/about/Stochastic_Finite_Elements.html?id=WzgKyTQQcAwC)] Ghanem, R. G., and Spanos, P. D. "Stochastic finite elements: a spectral approach", Springer New York, 1991. [**8500+**]

[[Xiu2002](https://epubs.siam.org/doi/pdf/10.1137/S1064827501387826)] Xiu, D., and Karniadakis, G. M. "The Wiener-Askey polynomial chaos for stochastic differential equations." *SIAM Journal on scientific computing* 24(2), 2002. [**5500+**]

[[Sudret2008](https://www.sciencedirect.com/science/article/pii/S0951832007001329)] Sudret, B. "Global sensitivity analysis using polynomial chaos expansions." *Reliability engineering & system safety* 93(7), 2008. [**2500+**]

[[Najm2009](https://www.annualreviews.org/doi/abs/10.1146/annurev.fluid.010908.165248)] Najm, H. N. "Uncertainty quantification and polynomial chaos techniques in computational fluid dynamics." *Annual review of fluid mechanics* 41, 2009. [**1000+**]

user: PieterjanRobbe
issue_27839: state: open
title: LocalOutlierFactor might not work with duplicated samples
body: This an investigation from the discussion in https://github.com/scikit-learn/scikit-learn/discussions/27838

`LocalFactorOutlier` might be difficult to use when there are duplicate values larger then `n_neighbors`. In this case, the distance for these neighbors is `0`, meaning that the local reachibility density is therefore infinite (or in the algorithm `1 / 1e-10`). The issue starts for sample next to those local peaky density: they might use the `1 / 1e-10` as measure, meaning that they will have a really negative `negative_local_outlier` while the value of the sample could be really close to the one of the plateau. I will now provide a minimum sythetic example to show the issue:

```python
import numpy as np
from sklearn.neighbors import LocalOutlierFactor

rng = np.random.default_rng(0)
x = rng.permutation(np.hstack([
    [0.1] * 10,  # constant values
    np.linspace(0.1, 0.2, num=30),
    rng.random(5) * 100  # the clear outliers
]))
X = x.reshape(-1, 1)

lof = LocalOutlierFactor(n_neighbors=5, contamination=0.1)
outliers = lof.fit_predict(X)

indices = np.where(outliers == -1)
# check that shows that outliers can be found from the linspace
print(X[indices])

print(lof.negative_outlier_factor_[indices])
```

```shell
array([[ 0.10344828],
       [26.97867138],
       [81.32702392],
       [63.69616873],
       [ 0.10689655]])

array([-3.31034492e+07, -1.22583005e+03, -1.10083976e+03, -9.37195958e+02,
       -4.13793114e+07])
```

In the results above, we see that the first and last values should not be considered as outliers but because they have their neighbors coming from the constant part (i.e. plateau at 0.1), the local reachibility density is 1e10 and thus the negative outlier factor is set to -1e7.

Running the same code without the constant part will give:

```python
import numpy as np
from sklearn.neighbors import LocalOutlierFactor

rng = np.random.default_rng(0)
x = rng.permutation(np.hstack([
    np.linspace(0.1, 0.2, num=30),
    rng.random(5) * 100  # the clear outliers
]))
X = x.reshape(-1, 1)

lof = LocalOutlierFactor(n_neighbors=5, contamination=0.1)
outliers = lof.fit_predict(X)

indices = np.where(outliers == -1)
# check that shows that outliers can be found from the linspace
print(X[indices])

print(lof.negative_outlier_factor_[indices])
```

```shell
[[81.32702392]
 [63.69616873]
 [ 4.09735239]
 [26.97867138]]
[-1100.83976485  -937.19595768  -237.40975167 -1225.83005019]
```

which is more what one would expect.

Now, my question would be if we can have a strategy to limit such a corner case that is ill-defined currently algorithmically? A potential solution is to find such extreme value and raise a warning and mentioning that there are duplicate and increasing `n_neighbors` could alleviate the problem?

ping @ngoix @albertcthomas @agramfort if you have any input.
user: glemaitre
issue_27836: state: open
title: [MRG] Implement Sparse Partial Least Squares (SPLS) in Cross Decomposition Module
body: #### What does this implement do? Explain your changes.

This pull request implements the Sparse Partial Least Squares (SPLS) algorithm in the sklearn.cross_decomposition module. SPLS extends the traditional PLS method by incorporating L1 regularization, enabling the model to handle high-dimensional data effectively by inducing sparsity in the weight vectors. This implementation includes:

- The SPLS algorithm with options to set the number of components and penalty parameters.

- Appropriate unit tests to ensure functionality and compliance with sklearn standards.

- Documentation updates, including a description of SPLS, its use cases, and a comparison with other PLS methods.

Any other comments?

The implementation deliberately follows the R package PMA [1], as authored by Witten, Tibshirani, and Hastie. Notably, SPLS has gained substantial attention in statistical and machine learning literature, often referenced as both a sparse PLS algorithm [3] and an approximation to sparse Canonical Correlation Analysis (CCA) [2] in high-dimensional contexts. The method's significance is underlined by its extensive citation, amassing nearly 1700 citations, indicating its wide acceptance and application across various domains [1].

Within the scope of scikit-learn, this implementation is positioned more clearly as a sparse PLS algorithm, following the soft thresholded power method approach. This addition is expected to enhance the toolkit's capabilities in advanced dimensionality reduction and feature selection, particularly for high-dimensional datasets.

Thanks for reviewing this contribution!

[1] Witten, D., Tibshirani, R., Gross, S., Narasimhan, B., & Witten, M. D. (2013). Package ‘pma’. Genetics and Molecular Biology, 8(1), 28.
[2] Witten, D. M., Tibshirani, R., & Hastie, T. (2009). A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. Biostatistics, 10(3), 515-534.
[2] Mihalik, A., Chapman, J., Adams, R. A., Winter, N. R., Ferreira, F. S., Shawe-Taylor, J., ... & Alzheimer’s Disease Neuroimaging Initiative. (2022). Canonical correlation analysis and partial least squares for identifying brain-behaviour associations: a tutorial and a comparative study. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging.
user: jameschapman19
issue_27825: state: open
title: MAINT apply assorted refurb suggestions
body: #### What does this implement/fix? Explain your changes.

Applies some refurb suggestions, only the ones that do improve readability.

#### Any other comments?

I know criteria of readability may depend on the reader. I am happy to remove changes that don't improve anything (possibly db5cbe15070de1281a9d40c56f8198d368b6e6a6). Alternatively you can of course disregard the whole PR :sweat_smile:
user: DimitriPapadopoulos
issue_27818: state: open
title: ENH Use scipy.stats.yeojohnson in PowerTransformer
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Closes #26308

#### What does this implement/fix? Explain your changes.
Use [scipy.stats.yeojohnson](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.yeojohnson.html#scipy-stats-yeojohnson) instead of our own implementation as @lorentzenchr suggested.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: xuefeng-xu
issue_27808: state: open
title: TransformedTargetRegressor with Early Stopping: transforming user-supplied validation sets in fit_params, too
body: ### Describe the workflow you want to enable

Many advanced regressors (CatBoost, XGBoost, LightGBM to name a few) support providing custom early stopping dataset(s) to their **fit** methods. Not all of them have scalar eval_fraction parameters, like GradientBoostingRegressor; even if they would, sometimes there is really a need for custom validation splitting (time-series with daily grouping, for instance).

Currently, TransformedTargetRegressor only transforms the **y** argument, which makes training with early stopping of above-mentioned regressors impossible: they do not converge, as I have just experienced. Or even if they converge, that can theoretically happen, validating on data in different scale can hardly find the same early stopping sweetspots.

### Describe your proposed solution

It would be convenient to support an extra **es_fit_param_name** argument to the init method of TransformedTargetRegressor.

When such param is present, at the fitting time,  fit_params are inspected for it.

```
if it's not in fit_params:
    error is raised
else:
   if it's an iterable of lists/tuples:
       self.transformer_.transform is additionally applied to the second element of each top-level iterable (fit_params will need modifying)
   elif it's an iterable:
       self.transformer_.transform is additionally applied to the second element of iterable (fit_params will need modifying)
   else:
       error is raised
```

Actually, for some most popular libraries relevant es_fit_param_name are known and can be used as defaults, if es_fit_param_name is not provided by the user but params nevertheless exist (can be turned off with an extra **no_es_autosuggest** flag).  But maybe going this deep is too much.

### Describe alternatives you've considered, if relevant

Applying transformer or func/inverse_func to the fit_params manually in every project using custom early stopping splitting.

### Additional context

A main drawback that I am seeing is the necessity to modify the user-provided fit_params, which is not a usual practice (but maybe we can copy fit_params internally first). On the other hand, it should allow thousands of data scientists to keep using both target transformations and custom early stopping in an elegant fashion. Ready to contribute if approved. Any thoughts? Is it worth it?
user: fingoldo
issue_27807: state: open
title: [MRG] Make random_state test for IterativeImputer behave as expected
body: #### Reference Issues/PRs
Fixes a test that was not implemented properly in #15636 (see also #15611).

#### What does this implement/fix? Explain your changes.
- #15636 added a test ensure IterativeImputer did not overwrite the random state of its estimator, but the test was not implemented correctly, and would trivially pass. This PR fixes that
- Updated note in the user guide on experimental status of IterativeImputer, to remove reference to an issue that's already closed (#15611)

#### Other comments
This is my first PR to an open source project, I apologise if I haven't done this properly.
user: leo-mazz
issue_27806: state: open
title: BUG: pytest error when loading conftest (seemingly platform-specific)
body: ### Describe the bug

I'm seeing errors on my Windows machine when running `pytest` (does not work with only `pytest`, and does not work for directories that has `conftest.py`). This seems to be a platform-specific problem, since CI is not complaining. I investigated a bit and found https://github.com/pytest-dev/pytest/issues/9765, but it doesn't look like `pytest` is planning to fix it, at least for now. I tried downgrading to `pytest==7.0.1` and everything worked smoothly, but in https://github.com/scikit-learn/scikit-learn/pull/26373 the minimum version of `pytest` has already been `7.1.2` for scikit-learn (due to some CI errors for `pytest==5.x.x`), so scikit-learn is raising error:

https://github.com/scikit-learn/scikit-learn/blob/5c4288dba42cb67d954cb56c2cebfbf25c05ef89/sklearn/conftest.py#L30-L34

I'm wondering if it is possible to pin `pytest==7.0.1` or at least relax the minimum requirement a bit to `PYTEST_MIN_VERSION = "7.0.1"`? Or are there any other suggestions how I may resolve this issue? @glemaitre who bumped the minimum version of `pytest` to 7.1.2. Truly sorry for the inconvenience caused by my annoying Windows machine.

### Steps/Code to Reproduce

```bash
pytest
```

or

```bash
pytest sklearn/utils/tests
```

Running `pytest` on a single file works correctly.

### Expected Results

No error is thrown.

### Actual Results

For the first example,

```pytb
❯ pytest
Traceback (most recent call last):
  File "D:\Downloads\mambaforge\envs\sklearn-env\Scripts\pytest-script.py", line 9, in <module>
    sys.exit(console_main())
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 192, in console_main
    code = main()
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 150, in main
    config = _prepareconfig(args, plugins)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 331, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_callers.py", line 130, in _multicall
    teardown[0].send(outcome)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\helpconfig.py", line 104, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 1075, in pytest_cmdline_parse
    self.parse(args)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 1425, in parse
    self._preparse(args, addopts=addopts)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 1327, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 1153, in pytest_load_initial_conftests
    self.pluginmanager._set_initial_conftests(
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 563, in _set_initial_conftests
    self._try_load_conftest(anchor, importmode, rootpath)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 585, in _try_load_conftest
    self._getconftestmodules(x, importmode, rootpath)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 609, in _getconftestmodules
    mod = self._importconftest(conftestpath, importmode, rootpath)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 654, in _importconftest
    assert mod not in mods
AssertionError
```

For the second example,

```pytb
❯ pytest sklearn/utils/tests
================================================================== test session starts ===================================================================
platform win32 -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: D:\ossd\scikit-learn-yxiao
configfile: setup.cfg
plugins: cov-4.1.0
collected 1957 items / 1 error

========================================================================= ERRORS =========================================================================
____________________________________________________ ERROR collecting sklearn/utils/tests/conftest.py ____________________________________________________
D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\runner.py:341: in from_call
    result: Optional[TResult] = func()
D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\doctest.py:560: in collect
    module = self.config.pluginmanager._importconftest(
D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py:654: in _importconftest
    assert mod not in mods
E   AssertionError
================================================================ short test summary info =================================================================
ERROR sklearn/utils/tests/conftest.py - AssertionError
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================== 1 warning, 1 error in 1.43s ===============================================================
```

### Versions

```shell
❯ python -c "import sklearn; sklearn.show_versions()"

System:
    python: 3.9.18 | packaged by conda-forge | (main, Aug 30 2023, 03:40:31) [MSC v.1929 64 bit (AMD64)]
executable: D:\Downloads\mambaforge\envs\sklearn-env\python.exe
   machine: Windows-10-10.0.19045-SP0

Python dependencies:
      sklearn: 1.4.dev0
          pip: 23.2.1
   setuptools: 68.2.2
        numpy: 1.26.0
        scipy: 1.11.2
       Cython: 3.0.2
       pandas: 2.1.1
   matplotlib: 3.8.0
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: mkl
    num_threads: 6
         prefix: libblas
       filepath: D:\Downloads\mambaforge\envs\sklearn-env\Library\bin\libblas.dll
        version: 2022.1-Product
threading_layer: intel

       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: vcomp
       filepath: D:\Downloads\mambaforge\envs\sklearn-env\vcomp140.dll
        version: None
```

user: Charlie-XIAO
issue_27800: state: open
title: FEA: Ridge support for Array API compliant inputs
body: #### What does this implement/fix? Explain your changes.

Follow-up to https://github.com/scikit-learn/scikit-learn/issues/26024

This PR is a WIP that aims at bringing support of Ridge for Array API comppliant inputs, starting with SVD solver.

user: fcharras
issue_27778: state: open
title: check_regressor_multioutput does not allow np.float32 predictions
body: ### Describe the bug

When testing my scikit-learn interface for my NN regressor, `check_regressor_multioutput` fails with an
`AssertionError: Multioutput predictions by a regressor are expected to be floating-point precision. Got float32 instead`
This is because the check only checks for `np.float64`. I can resolve this by casting the resulting array to `np.float64`, but then `check_methods_subset_invariance` fails instead because it applies a smaller tolerance for `float64` arrays.

Judging from the error message, I would guess that `np.float32` should be allowed in `check_regressor_multioutput`.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.base import BaseEstimator, RegressorMixin
from sklearn.utils.estimator_checks import check_regressor_multioutput


class MyRegressor(BaseEstimator, RegressorMixin):
    def fit(self, X, y):
        self.y_dim_ = y.shape[1]

    def predict(self, X):
        return np.zeros(shape=(X.shape[0], self.y_dim_), dtype=np.float32)


reg = MyRegressor()
check_regressor_multioutput('MyRegressor', reg)
```

### Expected Results

No error is thrown (at least not for this reason).

### Actual Results

```pytb
Traceback (most recent call last):
  File "bug_report_multioutput.py", line 15, in <module>
    check_regressor_multioutput('MyRegressor', reg)
  File "python3.10/site-packages/sklearn/utils/_testing.py", line 156, in wrapper
    return fn(*args, **kwargs)
  File "python3.10/site-packages/sklearn/utils/estimator_checks.py", line 2136, in check_regressor_multioutput
    assert y_pred.dtype == np.dtype("float64"), (
AssertionError: Multioutput predictions by a regressor are expected to be floating-point precision. Got float32 instead
```

### Versions

```shell
System:
    python: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
executable: /home/david/prog/venvs/tab_bench_venv/bin/python3.10
   machine: Linux-6.2.0-36-generic-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.3.2
          pip: 22.0.2
   setuptools: 59.6.0
        numpy: 1.24.3
        scipy: 1.10.1
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /media/data/prog/venvs/tab_bench_venv/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so
        version: 0.3.21
threading_layer: pthreads
   architecture: Zen
    num_threads: 16

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /media/data/prog/venvs/tab_bench_venv/lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 16

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /media/data/prog/venvs/tab_bench_venv/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Zen
    num_threads: 16
```

user: dholzmueller
issue_27777: state: open
title: HuberRegressor failed with ABNORMAL_TERMINATION_IN_LNSRCH on simple dataset
body: ### Describe the bug

sklearn linear_model HuberRegressor fails with a very simple dataset. 

The code below fails with ValueError: HuberRegressor convergence failed: l-BFGS-b solver terminated with ABNORMAL_TERMINATION_IN_LNSRCH



### Steps/Code to Reproduce

```python
import numpy as np
import sklearn
from sklearn.linear_model import HuberRegressor

X = np.array([10, 11, 28]).reshape(-1, 1)
y = np.log(np.array([5000., 5000., 5000.]))
sample_weights = np.array([2., 2., 2.])
h = HuberRegressor().fit(X, y, sample_weights)
```

### Expected Results

no ValueError thrown

### Actual Results

```pytb
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 10
      8 y = np.log(np.array([5000., 5000., 5000.]))
      9 sample_weights = np.array([2., 2., 2.])
---> 10 h = HuberRegressor().fit(X, y, sample_weights)

File ~\Anaconda3\lib\site-packages\sklearn\base.py:1151, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1144     estimator._validate_params()
   1146 with config_context(
   1147     skip_parameter_validation=(
   1148         prefer_skip_nested_validation or global_skip_validation
   1149     )
   1150 ):
-> 1151     return fit_method(estimator, *args, **kwargs)

File ~\Anaconda3\lib\site-packages\sklearn\linear_model\_huber.py:338, in HuberRegressor.fit(self, X, y, sample_weight)
    335 parameters = opt_res.x
    337 if opt_res.status == 2:
--> 338     raise ValueError(
    339         "HuberRegressor convergence failed: l-BFGS-b solver terminated with %s"
    340         % opt_res.message
    341     )
    342 self.n_iter_ = _check_optimize_result("lbfgs", opt_res, self.max_iter)
    343 self.scale_ = parameters[-1]

ValueError: HuberRegressor convergence failed: l-BFGS-b solver terminated with ABNORMAL_TERMINATION_IN_LNSRCH
```

### Versions

```shell
System:
    python: 3.9.18 (main, Sep 11 2023, 13:30:38) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\[username]\Anaconda3\python.exe
   machine: Windows-10-10.0.19044-SP0

Python dependencies:
      sklearn: 1.3.0
          pip: 23.3
   setuptools: 68.0.0
        numpy: 1.24.3
        scipy: 1.11.3
       Cython: None
       pandas: 2.1.1
   matplotlib: 3.8.0
       joblib: 1.2.0
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       filepath: C:\Users\[username]\Anaconda3\Library\bin\mkl_rt.2.dll
         prefix: mkl_rt
       user_api: blas
   internal_api: mkl
        version: 2023.1-Product
    num_threads: 4
threading_layer: intel

       filepath: C:\Users\[username]\Anaconda3\vcomp140.dll
         prefix: vcomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 8
```

user: jarilappalainen
issue_27767: state: open
title: EFF Reduce the size of shared objects of the C-extensions generated by Cython
body: ### Context

scikit-learn uses C-extensions in critical part of its implementations via Cython.

Each C-entension is build from one or several Cython translation unit (a `.pyx` file with a potential `.pxd` companion file). 

In scikit-learn, each C-extension build consists of a single Cython translation which is transpilled to a C or C++ translation unit, which is then compiled to a shared object file.

The resulting C or C++ translation unit contains the code translation from Cython to C and large preambule and epylogue of macros, functions, structs, global variables such as virtual tables, Python module definition, etc.

For instance, while the code of [`sklearn/utils/_heap.pyx`](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/_heap.pyx) only consists of less than 100 lines for a single function, the resulting `sklearn/utils/heap.c` file consists of more than 3500 lines, most of being the preambule's and the epilogue's injected by Cython:

<details>
<summary> Content of the generated <code>sklearn/utils/heap.c</code> </summary>

```

▾ macros
   -CYTHON_ABI
   -CYTHON_ASSUME_SAFE_MACROS
   -CYTHON_ASSUME_SAFE_MACROS
   -CYTHON_ASSUME_SAFE_MACROS
   -CYTHON_ASSUME_SAFE_MACROS
   -CYTHON_AVOID_BORROWED_REFS
   -CYTHON_AVOID_BORROWED_REFS
   -CYTHON_AVOID_BORROWED_REFS
   -CYTHON_AVOID_BORROWED_REFS
   -CYTHON_COMPILING_IN_CPYTHON
   -CYTHON_COMPILING_IN_CPYTHON
   -CYTHON_COMPILING_IN_CPYTHON
   -CYTHON_COMPILING_IN_CPYTHON
   -CYTHON_COMPILING_IN_NOGIL
   -CYTHON_COMPILING_IN_NOGIL
   -CYTHON_COMPILING_IN_NOGIL
   -CYTHON_COMPILING_IN_NOGIL
   -CYTHON_COMPILING_IN_PYPY
   -CYTHON_COMPILING_IN_PYPY
   -CYTHON_COMPILING_IN_PYPY
   -CYTHON_COMPILING_IN_PYPY
   -CYTHON_COMPILING_IN_PYSTON
   -CYTHON_COMPILING_IN_PYSTON
   -CYTHON_COMPILING_IN_PYSTON
   -CYTHON_COMPILING_IN_PYSTON
   -CYTHON_FALLTHROUGH
   -CYTHON_FALLTHROUGH
   -CYTHON_FALLTHROUGH
   -CYTHON_FALLTHROUGH
   -CYTHON_FALLTHROUGH
   -CYTHON_FALLTHROUGH
   -CYTHON_FAST_PYCALL
   -CYTHON_FAST_PYCALL
   -CYTHON_FAST_PYCALL
   -CYTHON_FAST_PYCALL
   -CYTHON_FAST_PYCCALL
   -CYTHON_FAST_THREAD_STATE
   -CYTHON_FAST_THREAD_STATE
   -CYTHON_FAST_THREAD_STATE
   -CYTHON_FAST_THREAD_STATE
   -CYTHON_FAST_THREAD_STATE
   -CYTHON_FORMAT_SSIZE_T
   -CYTHON_FUTURE_DIVISION
   -CYTHON_HEX_VERSION
   -CYTHON_INLINE
   -CYTHON_INLINE
   -CYTHON_INLINE
   -CYTHON_INLINE
   -CYTHON_INLINE
   -CYTHON_MAYBE_UNUSED_VAR(x)
   -CYTHON_NCP_UNUSED
   -CYTHON_NCP_UNUSED
   -CYTHON_PEP393_ENABLED
   -CYTHON_PEP393_ENABLED
   -CYTHON_PEP489_MULTI_PHASE_INIT
   -CYTHON_PEP489_MULTI_PHASE_INIT
   -CYTHON_PEP489_MULTI_PHASE_INIT
   -CYTHON_PEP489_MULTI_PHASE_INIT
   -CYTHON_PEP489_MULTI_PHASE_INIT
   -CYTHON_REFNANNY
   -CYTHON_RESTRICT
   -CYTHON_RESTRICT
   -CYTHON_RESTRICT
   -CYTHON_RESTRICT
   -CYTHON_SMALL_CODE
   -CYTHON_SMALL_CODE
   -CYTHON_SMALL_CODE
   -CYTHON_UNPACK_METHODS
   -CYTHON_UNPACK_METHODS
   -CYTHON_UNPACK_METHODS
   -CYTHON_UNPACK_METHODS
   -CYTHON_UNUSED
   -CYTHON_UNUSED
   -CYTHON_UNUSED
   -CYTHON_UNUSED
   -CYTHON_UPDATE_DESCRIPTOR_DOC
   -CYTHON_UPDATE_DESCRIPTOR_DOC
   -CYTHON_UPDATE_DESCRIPTOR_DOC
   -CYTHON_USE_ASYNC_SLOTS
   -CYTHON_USE_ASYNC_SLOTS
   -CYTHON_USE_ASYNC_SLOTS
   -CYTHON_USE_ASYNC_SLOTS
   -CYTHON_USE_ASYNC_SLOTS
   -CYTHON_USE_ASYNC_SLOTS
   -CYTHON_USE_DICT_VERSIONS
   -CYTHON_USE_DICT_VERSIONS
   -CYTHON_USE_DICT_VERSIONS
   -CYTHON_USE_DICT_VERSIONS
   -CYTHON_USE_EXC_INFO_STACK
   -CYTHON_USE_EXC_INFO_STACK
   -CYTHON_USE_EXC_INFO_STACK
   -CYTHON_USE_EXC_INFO_STACK
   -CYTHON_USE_EXC_INFO_STACK
   -CYTHON_USE_PYLIST_INTERNALS
   -CYTHON_USE_PYLIST_INTERNALS
   -CYTHON_USE_PYLIST_INTERNALS
   -CYTHON_USE_PYLIST_INTERNALS
   -CYTHON_USE_PYLONG_INTERNALS
   -CYTHON_USE_PYLONG_INTERNALS
   -CYTHON_USE_PYLONG_INTERNALS
   -CYTHON_USE_PYLONG_INTERNALS
   -CYTHON_USE_PYLONG_INTERNALS
   -CYTHON_USE_PYTYPE_LOOKUP
   -CYTHON_USE_PYTYPE_LOOKUP
   -CYTHON_USE_PYTYPE_LOOKUP
   -CYTHON_USE_PYTYPE_LOOKUP
   -CYTHON_USE_PYTYPE_LOOKUP
   -CYTHON_USE_TP_FINALIZE
   -CYTHON_USE_TP_FINALIZE
   -CYTHON_USE_TP_FINALIZE
   -CYTHON_USE_TP_FINALIZE
   -CYTHON_USE_TYPE_SLOTS
   -CYTHON_USE_TYPE_SLOTS
   -CYTHON_USE_TYPE_SLOTS
   -CYTHON_USE_TYPE_SLOTS
   -CYTHON_USE_UNICODE_INTERNALS
   -CYTHON_USE_UNICODE_INTERNALS
   -CYTHON_USE_UNICODE_INTERNALS
   -CYTHON_USE_UNICODE_INTERNALS
   -CYTHON_USE_UNICODE_WRITER
   -CYTHON_USE_UNICODE_WRITER
   -CYTHON_USE_UNICODE_WRITER
   -CYTHON_USE_UNICODE_WRITER
   -CYTHON_USE_UNICODE_WRITER
   -CYTHON_WITHOUT_ASSERTIONS
   -DL_EXPORT(t)
   -DL_IMPORT(t)
   -HAVE_LONG_LONG
   -METH_FASTCALL
   -METH_STACKLESS
   -PY_LONG_LONG
   -PY_SSIZE_T_CLEAN
   -PyBaseString_Type
   -PyBoolObject
   -PyByteArray_Check(obj)
   -PyIntObject
   -PyInt_AS_LONG
   -PyInt_AsLong
   -PyInt_AsSsize_t
   -PyInt_AsUnsignedLongLongMask
   -PyInt_AsUnsignedLongMask
   -PyInt_Check(op)
   -PyInt_CheckExact(op)
   -PyInt_FromLong
   -PyInt_FromSize_t
   -PyInt_FromSsize_t
   -PyInt_FromString
   -PyInt_FromUnicode
   -PyInt_Type
   -PyMem_RawFree(p)
   -PyMem_RawMalloc(n)
   -PyMem_RawRealloc(p,n)
   -PyNumber_Int
   -PyObject_ASCII(o)
   -PyObject_Format(obj,fmt)
   -PyObject_Free(p)
   -PyObject_Malloc(s)
   -PyObject_Realloc(p)
   -PyObject_Unicode
   -PySet_CheckExact(obj)
   -PyStringObject
   -PyString_Check
   -PyString_CheckExact
   -PyString_Type
   -PyUnicode_1BYTE_KIND
   -PyUnicode_2BYTE_KIND
   -PyUnicode_4BYTE_KIND
   -PyUnicode_Contains(u,s)
   -PyUnicode_InternFromString(s)
   -Py_BUILD_CORE
   -Py_HUGE_VAL
   -Py_TPFLAGS_CHECKTYPES
   -Py_TPFLAGS_HAVE_FINALIZE
   -Py_TPFLAGS_HAVE_INDEX
   -Py_TPFLAGS_HAVE_NEWBUFFER
   -Py_tss_NEEDS_INIT
   -_USE_MATH_DEFINES
   -__PYX_BUILD_PY_SSIZE_T
   -__PYX_COMMA
   -__PYX_DEFAULT_STRING_ENCODING
   -__PYX_DEFAULT_STRING_ENCODING_IS_ASCII
   -__PYX_DEFAULT_STRING_ENCODING_IS_DEFAULT
   -__PYX_DEFAULT_STRING_ENCODING_IS_UTF8
   -__PYX_DICT_VERSION_INIT
   -__PYX_ERR(f_index,lineno,Ln_error)
   -__PYX_EXTERN_C
   -__PYX_EXTERN_C
   -__PYX_GET_DICT_VERSION(dict)
   -__PYX_GET_DICT_VERSION(dict)
   -__PYX_HAVE_API__sklearn__utils___heap
   -__PYX_HAVE__sklearn__utils___heap
   -__PYX_MARK_ERR_POS(f_index,lineno)
   -__PYX_NAN()
   -__PYX_PY_DICT_LOOKUP_IF_MODIFIED(VAR,DICT,LOOKUP)
   -__PYX_PY_DICT_LOOKUP_IF_MODIFIED(VAR,DICT,LOOKUP)
   -__PYX_UPDATE_DICT_CACHE(dict,value,cache_var,version_var)
   -__PYX_UPDATE_DICT_CACHE(dict,value,cache_var,version_var)
   -__PYX_VERIFY_RETURN_INT(target_type,func_type,func_value)
   -__PYX_VERIFY_RETURN_INT_EXC(target_type,func_type,func_value)
   -__PYX__VERIFY_RETURN_INT(target_type,func_type,func_value,exc)
   -__Pyx_BUILTIN_MODULE_NAME
   -__Pyx_BUILTIN_MODULE_NAME
   -__Pyx_CLEAR(r)
   -__Pyx_CLineForTraceback(tstate,c_line)
   -__Pyx_DECREF(r)
   -__Pyx_DECREF(r)
   -__Pyx_DECREF_SET(r,v)
   -__Pyx_DefaultClassType
   -__Pyx_DefaultClassType
   -__Pyx_DefaultClassType
   -__Pyx_ErrFetch(type,value,tb)
   -__Pyx_ErrFetch(type,value,tb)
   -__Pyx_ErrFetchInState(tstate,type,value,tb)
   -__Pyx_ErrFetchWithState(type,value,tb)
   -__Pyx_ErrFetchWithState(type,value,tb)
   -__Pyx_ErrRestore(type,value,tb)
   -__Pyx_ErrRestore(type,value,tb)
   -__Pyx_ErrRestoreInState(tstate,type,value,tb)
   -__Pyx_ErrRestoreWithState(type,value,tb)
   -__Pyx_ErrRestoreWithState(type,value,tb)
   -__Pyx_GIVEREF(r)
   -__Pyx_GIVEREF(r)
   -__Pyx_GOTREF(r)
   -__Pyx_GOTREF(r)
   -__Pyx_HAS_GCC_DIAGNOSTIC
   -__Pyx_INCREF(r)
   -__Pyx_INCREF(r)
   -__Pyx_MODULE_NAME
   -__Pyx_NewRef(obj)
   -__Pyx_Owned_Py_None(b)
   -__Pyx_PyAsyncMethodsStruct
   -__Pyx_PyBaseString_Check(obj)
   -__Pyx_PyBaseString_Check(obj)
   -__Pyx_PyBaseString_CheckExact(obj)
   -__Pyx_PyBaseString_CheckExact(obj)
   -__Pyx_PyByteArray_FromCString(s)
   -__Pyx_PyByteArray_FromString(s)
   -__Pyx_PyByteArray_FromStringAndSize(s,l)
   -__Pyx_PyBytes_AsSString(s)
   -__Pyx_PyBytes_AsString(s)
   -__Pyx_PyBytes_AsUString(s)
   -__Pyx_PyBytes_AsWritableSString(s)
   -__Pyx_PyBytes_AsWritableString(s)
   -__Pyx_PyBytes_AsWritableUString(s)
   -__Pyx_PyBytes_FromCString(s)
   -__Pyx_PyBytes_FromString
   -__Pyx_PyBytes_FromStringAndSize
   -__Pyx_PyCFunctionFast
   -__Pyx_PyCFunctionFastWithKeywords
   -__Pyx_PyCode_HasFreeVars(co)
   -__Pyx_PyCode_HasFreeVars(co)
   -__Pyx_PyCode_New(a,k,l,s,f,code,c,n,v,fv,cell,fn,name,fline,lnos)
   -__Pyx_PyCode_New(a,k,l,s,f,code,c,n,v,fv,cell,fn,name,fline,lnos)
   -__Pyx_PyDict_GetItemStr(dict,name)
   -__Pyx_PyDict_GetItemStr(dict,name)
   -__Pyx_PyDict_NewPresized(n)
   -__Pyx_PyDict_NewPresized(n)
   -__Pyx_PyErr_Clear()
   -__Pyx_PyErr_Clear()
   -__Pyx_PyErr_GivenExceptionMatches(err,type)
   -__Pyx_PyErr_GivenExceptionMatches2(err,type1,type2)
   -__Pyx_PyErr_Occurred()
   -__Pyx_PyErr_Occurred()
   -__Pyx_PyErr_SetNone(exc)
   -__Pyx_PyErr_SetNone(exc)
   -__Pyx_PyErr_SetNone(exc)
   -__Pyx_PyException_Check(obj)
   -__Pyx_PyFastCFunction_Check(func)
   -__Pyx_PyFastCFunction_Check(func)
   -__Pyx_PyFrame_SetLineNumber(frame,lineno)
   -__Pyx_PyFrame_SetLineNumber(frame,lineno)
   -__Pyx_PyInt_AsHash_t
   -__Pyx_PyInt_AsHash_t
   -__Pyx_PyInt_FromHash_t
   -__Pyx_PyInt_FromHash_t
   -__Pyx_PyMODINIT_FUNC
   -__Pyx_PyMODINIT_FUNC
   -__Pyx_PyMODINIT_FUNC
   -__Pyx_PyMODINIT_FUNC
   -__Pyx_PyMODINIT_FUNC
   -__Pyx_PyMethod_New(func,self,klass)
   -__Pyx_PyMethod_New(func,self,klass)
   -__Pyx_PyNumber_Divide(x,y)
   -__Pyx_PyNumber_Divide(x,y)
   -__Pyx_PyNumber_Float(x)
   -__Pyx_PyNumber_InPlaceDivide(x,y)
   -__Pyx_PyNumber_InPlaceDivide(x,y)
   -__Pyx_PyNumber_Int(x)
   -__Pyx_PyNumber_Int(x)
   -__Pyx_PyObject_AsSString(s)
   -__Pyx_PyObject_AsUString(s)
   -__Pyx_PyObject_AsWritableSString(s)
   -__Pyx_PyObject_AsWritableString(s)
   -__Pyx_PyObject_AsWritableUString(s)
   -__Pyx_PyObject_FromCString(s)
   -__Pyx_PyObject_FromString
   -__Pyx_PyObject_FromStringAndSize
   -__Pyx_PyObject_GC_IsFinalized(o)
   -__Pyx_PyObject_GC_IsFinalized(o)
   -__Pyx_PyObject_GetAttrStr(o,n)
   -__Pyx_PySequence_SIZE(seq)
   -__Pyx_PySequence_SIZE(seq)
   -__Pyx_PySequence_Tuple(obj)
   -__Pyx_PyStr_FromCString(s)
   -__Pyx_PyStr_FromString
   -__Pyx_PyStr_FromString
   -__Pyx_PyStr_FromStringAndSize
   -__Pyx_PyStr_FromStringAndSize
   -__Pyx_PyString_Format(a,b)
   -__Pyx_PyString_Format(a,b)
   -__Pyx_PyString_FormatSafe(a,b)
   -__Pyx_PyThreadState_Current
   -__Pyx_PyThreadState_Current
   -__Pyx_PyThreadState_Current
   -__Pyx_PyThreadState_Current
   -__Pyx_PyThreadState_assign
   -__Pyx_PyThreadState_assign
   -__Pyx_PyThreadState_declare
   -__Pyx_PyThreadState_declare
   -__Pyx_PyType_AsAsync(obj)
   -__Pyx_PyType_AsAsync(obj)
   -__Pyx_PyType_AsAsync(obj)
   -__Pyx_PyUnicode_AsUnicode
   -__Pyx_PyUnicode_Concat(a,b)
   -__Pyx_PyUnicode_Concat(a,b)
   -__Pyx_PyUnicode_ConcatSafe(a,b)
   -__Pyx_PyUnicode_ConcatSafe(a,b)
   -__Pyx_PyUnicode_DATA(u)
   -__Pyx_PyUnicode_DATA(u)
   -__Pyx_PyUnicode_FormatSafe(a,b)
   -__Pyx_PyUnicode_FromCString(s)
   -__Pyx_PyUnicode_FromStringAndSize(c_str,size)
   -__Pyx_PyUnicode_FromStringAndSize(c_str,size)
   -__Pyx_PyUnicode_FromUnicode(u)
   -__Pyx_PyUnicode_FromUnicodeAndLength
   -__Pyx_PyUnicode_GET_LENGTH(u)
   -__Pyx_PyUnicode_GET_LENGTH(u)
   -__Pyx_PyUnicode_IS_TRUE(u)
   -__Pyx_PyUnicode_IS_TRUE(u)
   -__Pyx_PyUnicode_IS_TRUE(u)
   -__Pyx_PyUnicode_IS_TRUE(u)
   -__Pyx_PyUnicode_KIND(u)
   -__Pyx_PyUnicode_KIND(u)
   -__Pyx_PyUnicode_MAX_CHAR_VALUE(u)
   -__Pyx_PyUnicode_MAX_CHAR_VALUE(u)
   -__Pyx_PyUnicode_READ(k,d,i)
   -__Pyx_PyUnicode_READ(k,d,i)
   -__Pyx_PyUnicode_READY(op)
   -__Pyx_PyUnicode_READY(op)
   -__Pyx_PyUnicode_READY(op)
   -__Pyx_PyUnicode_READ_CHAR(u,i)
   -__Pyx_PyUnicode_READ_CHAR(u,i)
   -__Pyx_PyUnicode_WRITE(k,d,i,ch)
   -__Pyx_PyUnicode_WRITE(k,d,i,ch)
   -__Pyx_RefNannyDeclarations
   -__Pyx_RefNannyDeclarations
   -__Pyx_RefNannyFinishContext()
   -__Pyx_RefNannyFinishContext()
   -__Pyx_RefNannySetupContext(name,acquire_gil)
   -__Pyx_RefNannySetupContext(name,acquire_gil)
   -__Pyx_RefNannySetupContext(name,acquire_gil)
   -__Pyx_SET_REFCNT(obj,refcnt)
   -__Pyx_SET_REFCNT(obj,refcnt)
   -__Pyx_SET_SIZE(obj,size)
   -__Pyx_SET_SIZE(obj,size)
   -__Pyx_TypeCheck(obj,type)
   -__Pyx_TypeCheck(obj,type)
   -__Pyx_XCLEAR(r)
   -__Pyx_XDECREF(r)
   -__Pyx_XDECREF(r)
   -__Pyx_XDECREF_SET(r,v)
   -__Pyx_XGIVEREF(r)
   -__Pyx_XGIVEREF(r)
   -__Pyx_XGOTREF(r)
   -__Pyx_XGOTREF(r)
   -__Pyx_XINCREF(r)
   -__Pyx_XINCREF(r)
   -__Pyx_fits_Py_ssize_t(v,type,is_signed)
   -__Pyx_long_cast(x)
   -__Pyx_sst_abs(value)
   -__Pyx_sst_abs(value)
   -__Pyx_sst_abs(value)
   -__Pyx_sst_abs(value)
   -__Pyx_sst_abs(value)
   -__Pyx_sst_abs(value)
   -__Pyx_sst_abs(value)
   -__Pyx_truncl
   -__Pyx_truncl
   -__Pyx_uchar_cast(c)
   -__Pyx_void_to_None(void_result)
   -__cdecl
   -__fastcall
   -__has_attribute(x)
   -__has_cpp_attribute(x)
   -__pyx_PyFloat_AsDouble(x)
   -__pyx_PyFloat_AsDouble(x)
   -__pyx_PyFloat_AsFloat(x)
   -__stdcall
   -likely(x)
   -likely(x)
   -offsetof(type,member)
   -unlikely(x)
   -unlikely(x)

▾ prototypes
   -__Pyx_AddTraceback(const char * funcname,int c_line,int py_line,const char * filename)
   -__Pyx_CLineForTraceback(PyThreadState * tstate,int c_line)
   -__Pyx_ErrFetchInState(PyThreadState * tstate,PyObject ** type,PyObject ** value,PyObject ** tb)
   -__Pyx_ErrRestoreInState(PyThreadState * tstate,PyObject * type,PyObject * value,PyObject * tb)
   -__Pyx_ExportFunction(const char * name,void (* f)(void),const char * sig)
   -__Pyx_InitStrings(__Pyx_StringTabEntry * t)
   -__Pyx_IsSubtype(PyTypeObject * a,PyTypeObject * b)
   -__Pyx_PyBool_FromLong(long b)
   -__Pyx_PyErr_GivenExceptionMatches(PyObject * err,PyObject * type)
   -__Pyx_PyErr_GivenExceptionMatches2(PyObject * err,PyObject * type1,PyObject * type2)
   -__Pyx_PyIndex_AsHash_t(PyObject *)
   -__Pyx_PyIndex_AsSsize_t(PyObject *)
   -__Pyx_PyInt_As_int(PyObject *)
   -__Pyx_PyInt_As_long(PyObject *)
   -__Pyx_PyInt_FromSize_t(size_t)
   -__Pyx_PyInt_From_long(long value)
   -__Pyx_PyNumber_IntOrLong(PyObject * x)
   -__Pyx_PyObject_AsString(PyObject *)
   -__Pyx_PyObject_AsStringAndSize(PyObject *,Py_ssize_t * length)
   -__Pyx_PyObject_GetAttrStr(PyObject * obj,PyObject * attr_name)
   -__Pyx_PyObject_IsTrue(PyObject *)
   -__Pyx_PyObject_IsTrueAndDecref(PyObject *)
   -__Pyx_PyUnicode_FromString(const char *)
   -__Pyx_RefNannyImportAPI(const char * modname)
   -__Pyx_check_binary_version(void)
   -__Pyx_get_object_dict_version(PyObject * obj)
   -__Pyx_get_tp_dict_version(PyObject * obj)
   -__Pyx_modinit_function_export_code(void)
   -__Pyx_modinit_function_import_code(void)
   -__Pyx_modinit_global_init_code(void)
   -__Pyx_modinit_type_import_code(void)
   -__Pyx_modinit_type_init_code(void)
   -__Pyx_modinit_variable_export_code(void)
   -__Pyx_modinit_variable_import_code(void)
   -__Pyx_object_dict_version_matches(PyObject * obj,PY_UINT64_T tp_dict_version,PY_UINT64_T obj_dict_version)
   -__pyx_bisect_code_objects(__Pyx_CodeObjectCacheEntry * entries,int count,int code_line)
   -__pyx_find_code_object(int code_line)
   -__pyx_fuse_0__pyx_f_7sklearn_5utils_5_heap_heap_push(float *,__pyx_t_7sklearn_5utils_9_typedefs_intp_t *,__pyx_t_7sklearn_5utils_9_typedefs_intp_t,float,__pyx_t_7sklearn_5utils_9_typedefs_intp_t)
   -__pyx_fuse_1__pyx_f_7sklearn_5utils_5_heap_heap_push(double *,__pyx_t_7sklearn_5utils_9_typedefs_intp_t *,__pyx_t_7sklearn_5utils_9_typedefs_intp_t,double,__pyx_t_7sklearn_5utils_9_typedefs_intp_t)
   -__pyx_insert_code_object(int code_line,PyCodeObject * code_object)
   -__pyx_pymod_create(PyObject * spec,PyModuleDef * def)
   -__pyx_pymod_exec__heap(PyObject * module)
   -init_heap(void)

▾-__anonf7ac09720103 : enum
    [enumerators]
   +__pyx_check_sizeof_voidp

▾ typedefs
   -Py_hash_t
   -Py_tss_t
   -__Pyx_CodeObjectCacheEntry
   -__Pyx_PyAsyncMethodsStruct
   -__Pyx_PyCFunctionFast
   -__Pyx_PyCFunctionFastWithKeywords
   -__Pyx_RefNannyAPIStruct
   -__Pyx_StringTabEntry
   -__pyx_t_7sklearn_5utils_9_typedefs_float32_t
   -__pyx_t_7sklearn_5utils_9_typedefs_float64_t
   -__pyx_t_7sklearn_5utils_9_typedefs_int32_t
   -__pyx_t_7sklearn_5utils_9_typedefs_int64_t
   -__pyx_t_7sklearn_5utils_9_typedefs_intp_t
   -__pyx_t_7sklearn_5utils_9_typedefs_uint32_t
   -__pyx_t_7sklearn_5utils_9_typedefs_uint64_t
   -__pyx_t_7sklearn_5utils_9_typedefs_uint8_t
   -uint32_t
   -uint32_t
   -uint8_t
   -uint8_t

▾-__Pyx_CodeObjectCache : struct
    [members]
   +count
   +entries
   +max_count

▾-__anonf7ac09720208 : struct
    [members]
   +am_aiter
   +am_anext
   +am_await

▾-__anonf7ac09720308 : struct
    [members]
   +encoding
   +intern
   +is_str
   +is_unicode
   +n
   +p
   +s

▾-__anonf7ac09720408 : struct
    [members]
   +DECREF
   +FinishContext
   +GIVEREF
   +GOTREF
   +INCREF
   +SetupContext

▾-__anonf7ac09720508 : struct
    [members]
   +code_line
   +code_object

 -__anonf7ac0972060a : union

▾ variables
   -__PYX_DEFAULT_STRING_ENCODING
   -__Pyx_RefNanny
   -__Pyx_sys_getdefaultencoding_not_ascii
   -__pyx_b
   -__pyx_cfilenm
   -__pyx_clineno
   -__pyx_code_cache
   -__pyx_cython_runtime
   -__pyx_d
   -__pyx_empty_bytes
   -__pyx_empty_tuple
   -__pyx_empty_unicode
   -__pyx_f
   -__pyx_filename
   -__pyx_k_cline_in_traceback
   -__pyx_k_main
   -__pyx_k_name
   -__pyx_k_test
   -__pyx_lineno
   -__pyx_m
   -__pyx_methods
    __pyx_module_is_main_sklearn__utils___heap
   -__pyx_moduledef
   -__pyx_moduledef_slots
   -__pyx_n_s_cline_in_traceback
   -__pyx_n_s_main
   -__pyx_n_s_name
   -__pyx_n_s_test
   -__pyx_string_tab

▾ functions
    CYTHON_MAYBE_UNUSED_VAR(const T &)
   -PyThread_tss_alloc(void)
   -PyThread_tss_create(Py_tss_t * key)
   -PyThread_tss_delete(Py_tss_t * key)
   -PyThread_tss_free(Py_tss_t * key)
   -PyThread_tss_get(Py_tss_t * key)
   -PyThread_tss_is_created(Py_tss_t * key)
   -PyThread_tss_set(Py_tss_t * key,void * value)
   -__PYX_NAN()
   -__Pyx_AddTraceback(const char * funcname,int c_line,int py_line,const char * filename)
   -__Pyx_CLineForTraceback(CYTHON_UNUSED PyThreadState * tstate,int c_line)
   -__Pyx_CreateCodeObjectForTraceback(const char * funcname,int c_line,int py_line,const char * filename)
   -__Pyx_ErrFetchInState(PyThreadState * tstate,PyObject ** type,PyObject ** value,PyObject ** tb)
   -__Pyx_ErrRestoreInState(PyThreadState * tstate,PyObject * type,PyObject * value,PyObject * tb)
  ▾-__Pyx_ExportFunction(const char * name,void (* f)(void),const char * sig)
   -__Pyx_InBases(PyTypeObject * a,PyTypeObject * b)
   -__Pyx_InitCachedBuiltins(void)
   -__Pyx_InitCachedConstants(void)
   -__Pyx_InitGlobals(void)
   -__Pyx_InitStrings(__Pyx_StringTabEntry * t)
   -__Pyx_IsSubtype(PyTypeObject * a,PyTypeObject * b)
   -__Pyx_PyBool_FromLong(long b)
   -__Pyx_PyCode_New(int a,int k,int l,int s,int f,PyObject * code,PyObject * c,PyObject * n,PyObject * v,PyObject * fv,PyObject * cell,PyObject * fn,PyObject * name,int fline,PyObject * lnos)
   -__Pyx_PyErr_GivenExceptionMatches(PyObject * err,PyObject * exc_type)
   -__Pyx_PyErr_GivenExceptionMatches2(PyObject * err,PyObject * exc_type1,PyObject * exc_type2)
   -__Pyx_PyErr_GivenExceptionMatchesTuple(PyObject * exc_type,PyObject * tuple)
   -__Pyx_PyIndex_AsHash_t(PyObject * o)
   -__Pyx_PyIndex_AsSsize_t(PyObject * b)
   -__Pyx_PyInt_As_int(PyObject * x)
   -__Pyx_PyInt_As_long(PyObject * x)
   -__Pyx_PyInt_FromSize_t(size_t ival)
   -__Pyx_PyInt_From_long(long value)
   -__Pyx_PyNumber_IntOrLong(PyObject * x)
   -__Pyx_PyNumber_IntOrLongWrongResultType(PyObject * result,const char * type_name)
   -__Pyx_PyObject_AsString(PyObject * o)
   -__Pyx_PyObject_AsStringAndSize(PyObject * o,Py_ssize_t * length)
   -__Pyx_PyObject_GetAttrStr(PyObject * obj,PyObject * attr_name)
   -__Pyx_PyObject_IsTrue(PyObject * x)
   -__Pyx_PyObject_IsTrueAndDecref(PyObject * x)
   -__Pyx_PyUnicode_AsStringAndSize(PyObject * o,Py_ssize_t * length)
   -__Pyx_PyUnicode_AsStringAndSize(PyObject * o,Py_ssize_t * length)
   -__Pyx_PyUnicode_FromString(const char * c_str)
   -__Pyx_Py_UNICODE_strlen(const Py_UNICODE * u)
   -__Pyx_RefNannyImportAPI(const char * modname)
   -__Pyx_check_binary_version(void)
   -__Pyx_get_object_dict_version(PyObject * obj)
   -__Pyx_get_tp_dict_version(PyObject * obj)
   -__Pyx_init_sys_getdefaultencoding_params(void)
   -__Pyx_init_sys_getdefaultencoding_params(void)
   -__Pyx_inner_PyErr_GivenExceptionMatches2(PyObject * err,PyObject * exc_type1,PyObject * exc_type2)
   -__Pyx_inner_PyErr_GivenExceptionMatches2(PyObject * err,PyObject * exc_type1,PyObject * exc_type2)
   -__Pyx_is_valid_index(Py_ssize_t i,Py_ssize_t limit)
   -__Pyx_modinit_function_export_code(void)
   -__Pyx_modinit_function_import_code(void)
   -__Pyx_modinit_global_init_code(void)
   -__Pyx_modinit_type_import_code(void)
   -__Pyx_modinit_type_init_code(void)
   -__Pyx_modinit_variable_export_code(void)
   -__Pyx_modinit_variable_import_code(void)
   -__Pyx_object_dict_version_matches(PyObject * obj,PY_UINT64_T tp_dict_version,PY_UINT64_T obj_dict_version)
   -__Pyx_pretend_to_initialize(void * ptr)
   -__pyx_bisect_code_objects(__Pyx_CodeObjectCacheEntry * entries,int count,int code_line)
   -__pyx_find_code_object(int code_line)
   -__pyx_fuse_0__pyx_f_7sklearn_5utils_5_heap_heap_push(float * __pyx_v_values,__pyx_t_7sklearn_5utils_9_typedefs_intp_t * __pyx_v_indices,__pyx_t_7sklearn_5utils_9_typedefs_intp_t __pyx_v_size,float __pyx_v_val,__pyx_t_7sklearn_5utils_9_typedefs_intp_t __pyx_v_val_idx)
   -__pyx_fuse_1__pyx_f_7sklearn_5utils_5_heap_heap_push(double * __pyx_v_values,__pyx_t_7sklearn_5utils_9_typedefs_intp_t * __pyx_v_indices,__pyx_t_7sklearn_5utils_9_typedefs_intp_t __pyx_v_size,double __pyx_v_val,__pyx_t_7sklearn_5utils_9_typedefs_intp_t __pyx_v_val_idx)
   -__pyx_insert_code_object(int code_line,PyCodeObject * code_object)
    init_heap(void)
```

</details>

### Problem

Currently the uncompressed size of scikit-learn is around 48.8MB, 20MB of which are shared object files. As reported by @rth in https://github.com/pyodide/pyodide/issues/4289, while shared object files are optimized for Emscripten quite heavily, they still accounts for most of the size of scikit-learn on this stack.

<details>
<summary>Extensions' shared object sizes on Linux</summary>

```bash
find . -name \*.so -exec du -h {} \; | sort -h --reverse
```

```
2,1M	./sklearn/_loss/_loss.cpython-312-x86_64-linux-gnu.so
712K	./sklearn/utils/sparsefuncs_fast.cpython-312-x86_64-linux-gnu.so
672K	./sklearn/neighbors/_kd_tree.cpython-312-x86_64-linux-gnu.so
672K	./sklearn/neighbors/_ball_tree.cpython-312-x86_64-linux-gnu.so
624K	./sklearn/tree/_tree.cpython-312-x86_64-linux-gnu.so
608K	./sklearn/metrics/_dist_metrics.cpython-312-x86_64-linux-gnu.so
520K	./sklearn/datasets/_svmlight_format_fast.cpython-312-x86_64-linux-gnu.so
504K	./sklearn/preprocessing/_target_encoder_fast.cpython-312-x86_64-linux-gnu.so
476K	./sklearn/svm/_libsvm.cpython-312-x86_64-linux-gnu.so
448K	./sklearn/svm/_libsvm_sparse.cpython-312-x86_64-linux-gnu.so
448K	./sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.cpython-312-x86_64-linux-gnu.so
440K	./sklearn/metrics/_pairwise_distances_reduction/_datasets_pair.cpython-312-x86_64-linux-gnu.so
440K	./sklearn/linear_model/_cd_fast.cpython-312-x86_64-linux-gnu.so
404K	./sklearn/cluster/_k_means_elkan.cpython-312-x86_64-linux-gnu.so
396K	./sklearn/utils/_cython_blas.cpython-312-x86_64-linux-gnu.so
396K	./sklearn/cluster/_k_means_common.cpython-312-x86_64-linux-gnu.so
384K	./sklearn/preprocessing/_csr_polynomial_expansion.cpython-312-x86_64-linux-gnu.so
352K	./sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors.cpython-312-x86_64-linux-gnu.so
340K	./sklearn/tree/_splitter.cpython-312-x86_64-linux-gnu.so
340K	./sklearn/cluster/_hdbscan/_tree.cpython-312-x86_64-linux-gnu.so
324K	./sklearn/metrics/_pairwise_distances_reduction/_argkmin.cpython-312-x86_64-linux-gnu.so
324K	./sklearn/linear_model/_sgd_fast.cpython-312-x86_64-linux-gnu.so
312K	./sklearn/tree/_criterion.cpython-312-x86_64-linux-gnu.so
312K	./sklearn/cluster/_k_means_lloyd.cpython-312-x86_64-linux-gnu.so
308K	./sklearn/ensemble/_hist_gradient_boosting/splitting.cpython-312-x86_64-linux-gnu.so
308K	./sklearn/cluster/_hdbscan/_reachability.cpython-312-x86_64-linux-gnu.so
304K	./sklearn/cluster/_hierarchical_fast.cpython-312-x86_64-linux-gnu.so
292K	./sklearn/metrics/_pairwise_distances_reduction/_base.cpython-312-x86_64-linux-gnu.so
280K	./sklearn/linear_model/_sag_fast.cpython-312-x86_64-linux-gnu.so
276K	./sklearn/ensemble/_hist_gradient_boosting/histogram.cpython-312-x86_64-linux-gnu.so
272K	./sklearn/svm/_liblinear.cpython-312-x86_64-linux-gnu.so
268K	./sklearn/utils/_seq_dataset.cpython-312-x86_64-linux-gnu.so
264K	./sklearn/neighbors/_quad_tree.cpython-312-x86_64-linux-gnu.so
260K	./sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors_classmode.cpython-312-x86_64-linux-gnu.so
256K	./sklearn/_isotonic.cpython-312-x86_64-linux-gnu.so
252K	./sklearn/cluster/_k_means_minibatch.cpython-312-x86_64-linux-gnu.so
248K	./sklearn/utils/_fast_dict.cpython-312-x86_64-linux-gnu.so
248K	./sklearn/tree/_utils.cpython-312-x86_64-linux-gnu.so
248K	./sklearn/metrics/_pairwise_fast.cpython-312-x86_64-linux-gnu.so
244K	./sklearn/decomposition/_online_lda_fast.cpython-312-x86_64-linux-gnu.so
240K	./sklearn/metrics/_pairwise_distances_reduction/_argkmin_classmode.cpython-312-x86_64-linux-gnu.so
232K	./sklearn/utils/_typedefs.cpython-312-x86_64-linux-gnu.so
232K	./sklearn/utils/_isfinite.cpython-312-x86_64-linux-gnu.so
220K	./sklearn/utils/arrayfuncs.cpython-312-x86_64-linux-gnu.so
220K	./sklearn/ensemble/_gradient_boosting.cpython-312-x86_64-linux-gnu.so
216K	./sklearn/ensemble/_hist_gradient_boosting/utils.cpython-312-x86_64-linux-gnu.so
212K	./sklearn/utils/_random.cpython-312-x86_64-linux-gnu.so
212K	./sklearn/utils/murmurhash.cpython-312-x86_64-linux-gnu.so
212K	./sklearn/ensemble/_hist_gradient_boosting/_predictor.cpython-312-x86_64-linux-gnu.so
212K	./sklearn/decomposition/_cdnmf_fast.cpython-312-x86_64-linux-gnu.so
212K	./sklearn/cluster/_hdbscan/_linkage.cpython-312-x86_64-linux-gnu.so
204K	./sklearn/metrics/cluster/_expected_mutual_info_fast.cpython-312-x86_64-linux-gnu.so
204K	./sklearn/manifold/_barnes_hut_tsne.cpython-312-x86_64-linux-gnu.so
192K	./sklearn/utils/_weight_vector.cpython-312-x86_64-linux-gnu.so
184K	./sklearn/manifold/_utils.cpython-312-x86_64-linux-gnu.so
184K	./sklearn/ensemble/_hist_gradient_boosting/_gradient_boosting.cpython-312-x86_64-linux-gnu.so
184K	./sklearn/cluster/_dbscan_inner.cpython-312-x86_64-linux-gnu.so
180K	./sklearn/ensemble/_hist_gradient_boosting/_bitset.cpython-312-x86_64-linux-gnu.so
180K	./sklearn/ensemble/_hist_gradient_boosting/_binning.cpython-312-x86_64-linux-gnu.so
128K	./sklearn/utils/_vector_sentinel.cpython-312-x86_64-linux-gnu.so
112K	./sklearn/ensemble/_hist_gradient_boosting/common.cpython-312-x86_64-linux-gnu.so
80K	    ./sklearn/feature_extraction/_hashing_fast.cpython-312-x86_64-linux-gnu.so
48K	    ./sklearn/utils/_openmp_helpers.cpython-312-x86_64-linux-gnu.so
32K	    ./sklearn/svm/_newrand.cpython-312-x86_64-linux-gnu.so
28K	    ./sklearn/utils/_sorting.cpython-312-x86_64-linux-gnu.so
28K	    ./sklearn/neighbors/_partition_nodes.cpython-312-x86_64-linux-gnu.so
24K	    ./sklearn/utils/_heap.cpython-312-x86_64-linux-gnu.so
24K	    ./sklearn/__check_build/_check_build.cpython-312-x86_64-linux-gnu.so
```

</details>

### Possible solutions

#### Strip all symbols and optimize for size

This can be done by adding `-Wl,--strip-all` to `extra_link_args` and `-Os -g0` to `extra_compile_args`.

In practice, it can significantly shrink shared object (up to nearly 50% size reduction):

<details>

<summary>Extensions' shared object sizes on Linux after striping all symbols and optimizing for size</summary>

```
find . -name \*.so -exec du -h {} \; | sort -h --reverse
```

```
1,2M	sklearn/_loss/_loss.cpython-312-x86_64-linux-gnu.so
560K	sklearn/tree/_tree.cpython-312-x86_64-linux-gnu.so
468K	sklearn/neighbors/_kd_tree.cpython-312-x86_64-linux-gnu.so
468K	sklearn/neighbors/_ball_tree.cpython-312-x86_64-linux-gnu.so
460K	sklearn/utils/sparsefuncs_fast.cpython-312-x86_64-linux-gnu.so
460K	sklearn/metrics/_dist_metrics.cpython-312-x86_64-linux-gnu.so
380K	sklearn/datasets/_svmlight_format_fast.cpython-312-x86_64-linux-gnu.so
376K	sklearn/svm/_libsvm.cpython-312-x86_64-linux-gnu.so
352K	sklearn/svm/_libsvm_sparse.cpython-312-x86_64-linux-gnu.so
336K	sklearn/tree/_splitter.cpython-312-x86_64-linux-gnu.so
320K	sklearn/preprocessing/_target_encoder_fast.cpython-312-x86_64-linux-gnu.so
320K	sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.cpython-312-x86_64-linux-gnu.so
316K	sklearn/metrics/_pairwise_distances_reduction/_datasets_pair.cpython-312-x86_64-linux-gnu.so
312K	sklearn/utils/_cython_blas.cpython-312-x86_64-linux-gnu.so
308K	sklearn/linear_model/_cd_fast.cpython-312-x86_64-linux-gnu.so
300K	sklearn/tree/_criterion.cpython-312-x86_64-linux-gnu.so
280K	sklearn/preprocessing/_csr_polynomial_expansion.cpython-312-x86_64-linux-gnu.so
280K	sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors.cpython-312-x86_64-linux-gnu.so
280K	sklearn/cluster/_k_means_elkan.cpython-312-x86_64-linux-gnu.so
276K	sklearn/cluster/_k_means_common.cpython-312-x86_64-linux-gnu.so
252K	sklearn/linear_model/_sgd_fast.cpython-312-x86_64-linux-gnu.so
248K	sklearn/cluster/_hdbscan/_tree.cpython-312-x86_64-linux-gnu.so
244K	sklearn/metrics/_pairwise_distances_reduction/_argkmin.cpython-312-x86_64-linux-gnu.so
240K	sklearn/tree/_utils.cpython-312-x86_64-linux-gnu.so
240K	sklearn/ensemble/_hist_gradient_boosting/splitting.cpython-312-x86_64-linux-gnu.so
236K	sklearn/cluster/_hierarchical_fast.cpython-312-x86_64-linux-gnu.so
236K	sklearn/cluster/_hdbscan/_reachability.cpython-312-x86_64-linux-gnu.so
232K	sklearn/svm/_liblinear.cpython-312-x86_64-linux-gnu.so
228K	sklearn/cluster/_k_means_lloyd.cpython-312-x86_64-linux-gnu.so
224K	sklearn/metrics/_pairwise_distances_reduction/_base.cpython-312-x86_64-linux-gnu.so
212K	sklearn/neighbors/_quad_tree.cpython-312-x86_64-linux-gnu.so
208K	sklearn/linear_model/_sag_fast.cpython-312-x86_64-linux-gnu.so
204K	sklearn/ensemble/_hist_gradient_boosting/histogram.cpython-312-x86_64-linux-gnu.so
200K	sklearn/utils/_seq_dataset.cpython-312-x86_64-linux-gnu.so
196K	sklearn/utils/_fast_dict.cpython-312-x86_64-linux-gnu.so
196K	sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors_classmode.cpython-312-x86_64-linux-gnu.so
196K	sklearn/_isotonic.cpython-312-x86_64-linux-gnu.so
196K	sklearn/decomposition/_online_lda_fast.cpython-312-x86_64-linux-gnu.so
196K	sklearn/cluster/_k_means_minibatch.cpython-312-x86_64-linux-gnu.so
192K	sklearn/utils/_isfinite.cpython-312-x86_64-linux-gnu.so
192K	sklearn/metrics/_pairwise_fast.cpython-312-x86_64-linux-gnu.so
192K	sklearn/metrics/_pairwise_distances_reduction/_argkmin_classmode.cpython-312-x86_64-linux-gnu.so
184K	sklearn/utils/_typedefs.cpython-312-x86_64-linux-gnu.so
180K	sklearn/utils/arrayfuncs.cpython-312-x86_64-linux-gnu.so
180K	sklearn/ensemble/_hist_gradient_boosting/utils.cpython-312-x86_64-linux-gnu.so
180K	sklearn/ensemble/_gradient_boosting.cpython-312-x86_64-linux-gnu.so
172K	sklearn/utils/_random.cpython-312-x86_64-linux-gnu.so
168K	sklearn/utils/murmurhash.cpython-312-x86_64-linux-gnu.so
168K	sklearn/ensemble/_hist_gradient_boosting/_predictor.cpython-312-x86_64-linux-gnu.so
168K	sklearn/decomposition/_cdnmf_fast.cpython-312-x86_64-linux-gnu.so
168K	sklearn/cluster/_hdbscan/_linkage.cpython-312-x86_64-linux-gnu.so
160K	sklearn/metrics/cluster/_expected_mutual_info_fast.cpython-312-x86_64-linux-gnu.so
160K	sklearn/manifold/_barnes_hut_tsne.cpython-312-x86_64-linux-gnu.so
152K	sklearn/utils/_weight_vector.cpython-312-x86_64-linux-gnu.so
152K	sklearn/manifold/_utils.cpython-312-x86_64-linux-gnu.so
152K	sklearn/ensemble/_hist_gradient_boosting/_gradient_boosting.cpython-312-x86_64-linux-gnu.so
152K	sklearn/cluster/_dbscan_inner.cpython-312-x86_64-linux-gnu.so
148K	sklearn/ensemble/_hist_gradient_boosting/_bitset.cpython-312-x86_64-linux-gnu.so
144K	sklearn/ensemble/_hist_gradient_boosting/_binning.cpython-312-x86_64-linux-gnu.so
112K	sklearn/utils/_vector_sentinel.cpython-312-x86_64-linux-gnu.so
96K	sklearn/ensemble/_hist_gradient_boosting/common.cpython-312-x86_64-linux-gnu.so
68K	sklearn/feature_extraction/_hashing_fast.cpython-312-x86_64-linux-gnu.so
44K	sklearn/utils/_openmp_helpers.cpython-312-x86_64-linux-gnu.so
32K	sklearn/svm/_newrand.cpython-312-x86_64-linux-gnu.so
28K	sklearn/utils/_sorting.cpython-312-x86_64-linux-gnu.so
28K	sklearn/neighbors/_partition_nodes.cpython-312-x86_64-linux-gnu.so
24K	sklearn/utils/_heap.cpython-312-x86_64-linux-gnu.so
24K	sklearn/__check_build/_check_build.cpython-312-x86_64-linux-gnu.so
```
</details>

#### Group several translation units within C extensions (and use interprocedural optimization)

So as to reuse duplicated symbols in shared objects and perform optimization over several translation units (such as inlining functions, etc.)
user: jjerphan
issue_27763: state: open
title: FIX handle properly null weights in SVC
body: closes https://github.com/scikit-learn/scikit-learn/issues/5150
closes https://github.com/scikit-learn/scikit-learn/issues/25380

This changes intends to:

- [x] fix the indexing of the support vector `support_` since negative and zeros samples are removed
- [x] fix the value reported by `n_support_`
- [ ] fix the dual coefficient when the removing sample via sample weights are cancelling an entire class
- bonus trying to make the code more readable by removing single letter variable
user: glemaitre
issue_27751: state: open
title: Groups in BaggingRegressor
body: ### Describe the workflow you want to enable

It would be nice if we could control how random indices are created with groups in BaggingRegressor. As far as I can tell, BaggingRegressor will select sub samples u.a.r. with/without replacement.

It would be nice if we could group the data points, and have BaggingRegressor select sub samples of (optional) _groups_ u.a.r. with/without replacement.

An example of this paradigm already exists in sklearn with cross validation (e.g. KFold vs. GroupKFold)

### Describe your proposed solution

One idea is to pass in a vector `groups` of size `len(X)` mapping each data point to its group. Then, `_generate_indices` can respect groups if they exist, or default to its existing behavior if they do not.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: neil-kulkarni
labels: Needs Decision: Requires decision
issue_27749: state: open
title: Add handle_missing and handle_unknown options to TargetEncoder
body: ### Describe the workflow you want to enable

This issue has a similar proposition then #17123, in which is discussed the addition of `handle_missing` and `handle_unknown` from `OrdinalEncoder`  in  `category_encoders` library. The justification for it is that we should allow users to treat the NaN values as they wish, whether by imputation, algorithms robust to missing data or by any another approach. 

To see if this change can have a positive impact on modeling, I decided to run a Cross Validation in order to compare the performance of `category_encoders.target_encoder.TargetEncoder` allowing to return NaN values ​​and allowing to return a value (target mean).

```python

from ucimlrepo import fetch_ucirepo 
from sklearn.ensemble import BaggingClassifier
from sklearn.pipeline import Pipeline
from category_encoders import TargetEncoder
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold
import matplotlib.pyplot as plt
#%%

adult = fetch_ucirepo(id=2) 
  
X = adult.data.features 
y = adult.data.targets 

y = y['income'].str.contains('>50K')

# %%

kf = StratifiedKFold(5, shuffle = True, random_state = 101)

return_nan_clf = Pipeline([
    ('encoder', TargetEncoder(handle_unknown='return_nan',
                                  handle_missing='return_nan')),
    ('classifier', BaggingClassifier(
        DecisionTreeClassifier(
            max_features = 'sqrt'
            ),
        n_estimators=100,
        
        ))
    ])

mean_clf = Pipeline([
    ('encoder', TargetEncoder(handle_unknown='value',
                                  handle_missing='value')),
    ('classifier', BaggingClassifier(
        DecisionTreeClassifier(
            max_features = 'sqrt'
            ),
        n_estimators=100,
        ))
    ])

# %%

return_nan_cv = cross_val_score(return_nan_clf, X, y, cv = kf, verbose = 3,
                                scoring = 'f1')

print(f'\nAllowing Missing Data CV F1 score: {return_nan_cv.mean():.3f}\n')

mean_cv = cross_val_score(mean_clf, X, y, cv = kf, verbose = 3,
                          scoring = 'f1')

print(f'\nTreating Missing Data CV F1 score: {mean_cv.mean():.3f}')

# %%

fig, ax = plt.subplots(1, 1, figsize = (12, 6))

diff_cv = return_nan_cv - mean_cv

colors = ['b', 'b', 'r', 'b', 'b']

ax.set_title('Difference between allowing NaN values and treating them during encoding',
             fontsize = 15)
ax.bar(range(1, diff_cv.size + 1), diff_cv, color = colors)
ax.set_ylim(-0.002, 0.010)
ax.set_ylabel('F1 Score', fontsize = 15)
ax.set_xlabel('Folds', fontsize = 15)
ax.tick_params(axis='both', which='major', labelsize=12)
``` 
![image](https://github.com/scikit-learn/scikit-learn/assets/34726102/6ec103a9-01bf-466d-a47c-2b7fd76d8253)

As we can clearly see, allowing NaN values resulted in an increase in performance in most Folds. 


### Describe your proposed solution

Adapt the  `handle_missing` and `handle_unknown` options from `category_encoders.target_encoder.TargetEncoder` to `sklearn.preprocessing.TargetEncoder`.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: hnrosa
issue_27745: state: open
title: Tutorial on how to create a custom kernel for Gaussian Process
body: #### Reference Issues/PRs
Fixes #9601. Continues the work of  #17642 and #9605


#### What does this implement/fix? Explain your changes.
Took the code  and integrating comments from  #17642, and put it in examples folder. About custom kernel for gaussian process

#### Any other comments?


@betatim @glemaitre 

user: margot95
issue_27743: state: open
title: Devcontainer configuration added
body: #### Reference Issues/PRs
None

#### What does this implement/fix? 

At a scikit-learn sprint many attendees have problems getting started with contributing and setting up their environment. Be it to clone a the repo, installing dependencies, ...

The github codespaces as an alternative are a possibility to speed this part up, such that most time can be used going through code/ updating docs.


#### Any other comments? 

If this is even interesting or should be supported can be debated. 

At least for updating docs it might be simpler then having people go through the contributing guide. 

This PR probably is only helpful to people contributing once or a few times at most.

user: brunnelu
issue_27741: state: open
title: Make an instance of ColumnTransformer pass the common test
body: ### Describe the bug

Not sure if `check_estimator` is appropriate to use with transformers, but `ColumnTransformer` inherits from `BaseEstimator`, so I'm assuming so.

I have a custom transformer that inherits from `ColumnTransformer`, and I'm trying to use `check_estimator` to make sure I am doing so properly. However, `ColumnTransformer` itself fails on multiple tests; I'm assuming many are because creating a non-trivial `ColumnTransformer` involves specifying required feature names, which seems to go against some of the tests in `check_estimator`.

Is there a different best practice to test a custom subclass of `ColumnTransformer` and/or transformers in general?

### Steps/Code to Reproduce

```python
from sklearn.utils.estimator_checks import parametrize_with_checks
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import Normalizer

ct = ColumnTransformer(
    [
        ("norm1", Normalizer(norm="l1"), [0, 1]),
        ("norm2", Normalizer(norm="l1"), slice(2, 4)),
    ],
    remainder="passthrough",
)


@parametrize_with_checks([ct])
def test_sklearn_checks(estimator, check):
    check(estimator)
```

### Expected Results

All tests pass.

### Actual Results

[Test Results —.pdf](https://github.com/scikit-learn/scikit-learn/files/13285011/Test.Results.pdf)


### Versions

```shell
System:
    python: 3.11.6 | packaged by conda-forge | (main, Oct  3 2023, 10:37:07) [Clang 15.0.7 ]
executable: /my_env/bin/python
   machine: macOS-13.4-arm64-arm-64bit
Python dependencies:
      sklearn: 1.3.1
          pip: 23.3.1
   setuptools: 68.2.2
        numpy: 1.26.0
        scipy: 1.11.3
       Cython: None
       pandas: 2.1.1
   matplotlib: 3.8.0
       joblib: 1.3.2
threadpoolctl: 3.2.0
Built with OpenMP: True
threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /my_env/lib/libopenblas.0.dylib
        version: 0.3.24
threading_layer: openmp
   architecture: VORTEX
       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /my_env/lib/libomp.dylib
        version: None
```

user: rjboczar
issue_27740: state: open
title: HalvingGridSearchCV should not care about parameter-grid layout but apparently does
body: ### Describe the bug

I have two parameter-grid layouts, both specifying the exact same set of configurations. The difference is that the choices are listed in a different order, e.g., [False, True] versus [True, False]. My understanding is that this should not matter given that HalvingGridSearchCV tries all configurations on the first step. However, I get different results when using first grid versus the second.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.svm import SVC
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import HalvingGridSearchCV

#three grids with identical configuration space

svc_grid_1 = dict(C=[1,2,3],
                gamma=['auto', 'scale'],
                shrinking=(True, False),
                kernel=['sigmoid', 'linear', 'poly', 'rbf'],
                max_iter = [5000, 10000, -1]
                )

svc_grid_2 = dict(C=[1,2,3],
                gamma=['auto', 'scale'],
                shrinking=(True, False),
                kernel=['rbf', 'poly', 'linear',  'sigmoid'],  #reordered choices
                max_iter = [5000, 10000, -1]
                )

svc_grid_3 = dict(kernel=['sigmoid', 'linear', 'poly', 'rbf'],  #reordered keys
                  C=[1,2,3],
                  shrinking=(True, False),
                  max_iter = [5000, 10000, -1],
                  gamma=['auto', 'scale'],
                )

###How many different combinations?

from sklearn.model_selection import ParameterGrid
param_grid_1 = ParameterGrid(svc_grid_1)  #a list of dictionaries, one for each combo
print(f'{len(param_grid_1)=}')  #144
param_grid_2 = ParameterGrid(svc_grid_2)  #a list of dictionaries, one for each combo
print(f'{len(param_grid_1)=}')  #144
param_grid_3 = ParameterGrid(svc_grid_3)  #a list of dictionaries, one for each combo
print(f'{len(param_grid_1)=}')  #144

all([d in param_grid_2 and d in param_grid_3 for d in param_grid_1])  #True

#random data

np.random.seed(0)
n_samples, n_features = 1313, 6
x_train = np.random.rand(n_samples, n_features)
y_train = np.random.randint(0, 2, n_samples)  # binary target with values in {0, 1}

#Try Grid 1

svc_model = SVC(probability=True, random_state=1)  #base model
min_res = 30

#do the grid search
halving_cv = HalvingGridSearchCV(
    svc_model, svc_grid_1,  
    scoring="roc_auc",  
    n_jobs=1,  
    min_resources=min_res,
    factor=2,  
    cv=5, random_state=1234,
    refit=True,  
)

grid_result_1 = halving_cv.fit(x_train, y_train)
grid_result_1.best_params_ 
#{'C': 1,
# 'gamma': 'scale',
# 'kernel': 'sigmoid',
# 'max_iter': -1,
# 'shrinking': True}

#Try grid 2

svc_model = SVC(probability=True, random_state=1)  #base model

min_res = 30

#do the grid search
halving_cv = HalvingGridSearchCV(
    svc_model, svc_grid_2,  
    scoring="roc_auc", 
    n_jobs=1,  
    min_resources=min_res,
    factor=2, 
    cv=5, random_state=1234,
    refit=True, 
)

grid_result_2 = halving_cv.fit(x_train, y_train)
grid_result_2.best_params_ 
#{'C': 1,
 #'gamma': 'scale',
 #'kernel': 'sigmoid',
 #'max_iter': 5000,
 #'shrinking': True}

set(grid_result_1.best_params_.values()) == set(grid_result_2.best_params_.values())  #False
```
### Expected Results

best_params_  would be same for both grids

### Actual Results

They are different.

### Versions

```shell
System:
    python: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
executable: /usr/bin/python3
   machine: Linux-5.15.120+-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.2
   setuptools: 67.7.2
        numpy: 1.23.5
        scipy: 1.11.3
       Cython: 3.0.4
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/numpy.libs/libopenblas64_p-r0-742d56dc.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libgomp
       filepath: /usr/local/lib/python3.10/dist-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Haswell
```

user: fickas
labels: Needs Investigation: Issue requires investigation
issue_27736: state: open
title: ENH Add Array API compatibility to `mean_absolute_error`
body: #### Reference Issues/PRs

Towards https://github.com/scikit-learn/scikit-learn/issues/26024

#### What does this implement/fix? Explain your changes.

It makes the `mean_absolute_error` implementation compatible and tested with the Array API.

Not sure if it's the best approach, but I have converted the `np.average` implementation so that it is compatible with the Array API. Is there a better way? (I will fix add the tests to make codecov happy, if you agree to have the `_average` function)

cc @betatim @ogrisel 
user: EdAbati
issue_27724: state: open
title: FIX make SparseCoder pass the common tests
body: closes #26482
closes #26691
superseded #26691

While reviewing #26691, we saw that there is a coupling between the parameter `dictionary` in `SparseCoder` and `X`. Therefore, the simplest way to not complicate the common tests is to run them at the level of the module and inherit the `SparseCoder` to magically set the `dictionary` attribute.
user: glemaitre
issue_27722: state: open
title: FEA add SelectThreshold to filter score based on user-specified threshold
body: Depends on https://github.com/scikit-learn/scikit-learn/pull/27721
closes https://github.com/scikit-learn/scikit-learn/issues/27609

Add `SelectThreshold` that is a filter selecting features that have a higher score than a user-defined threshold.

This selector allows for similar use case to `VarianceThreshold` but has more flexibility since the score function is not limited to the variance.
user: glemaitre
issue_27713: state: open
title: ENH Add rich progress bar while downloading
body: #### Reference Issues/PRs

Fixes #27592
Related PR #27663


#### What does this implement/fix? Explain your changes.

I implemented progress bar while fetching datasets using `rich` library

#### Any other comments?

Currently, this PR is WIP, and only works with the function `fetch_lfw_people` and `fetch_california_housing`. The remaining functions are not implemented, so I will implement the remaining functions after receiving a review.

**Known gliches**
If we use the console output (ex.print) after using the rich progress bar, there is a problem that the terminal cursor operation of rich outputs abnormally.

<img width="510" alt="image" src="https://github.com/scikit-learn/scikit-learn/assets/25397908/d2167861-0a7b-4e94-9e0b-933ff26ca2ec">


Here are some sample (test) code for this PR:
```python
from sklearn.datasets._lfw import fetch_lfw_people
from sklearn.datasets._california_housing import fetch_california_housing
from sklearn.datasets import clear_data_home
clear_data_home()
print('Testing lfw..')
lfw_people = fetch_lfw_people(min_faces_per_person=70, resize=0.4, 
                              show_progress=True)
print("Done!")
clear_data_home()

print('Testing california housing..')
california = fetch_california_housing(show_progress=True)
print("Done!")
```

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: mirusu400
issue_27709: state: open
title: ENH support mutlilabel classification with VotingClassifier
body: close #19223
partially address #18289

This PR intends to bring support for multilabel classification in `VotingClassifier`.

In draft until merging:

- https://github.com/scikit-learn/scikit-learn/pull/27710
- https://github.com/scikit-learn/scikit-learn/pull/27702
user: glemaitre
issue_27704: state: open
title: [MRG] ENH: Make StackingRegressor support Multioutput
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Related to #25597
Similar to #8547
Similar to #19223

#### What does this implement/fix? Explain your changes.

- Added the support for multioutput in StackingRegressor;
- Added the test codes for above changes.
- Update the docstring of StackingRegressor.

#### Any other comments?
I am concerned the followings:
- Do we need any other tests?

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: hmasdev
issue_27676: state: open
title: Callback API plan
body: The goal of this issue is to track the steps of the implementation of a callback API in scikit-learn.

This is being developed in the `callbacks` feature branch. The first PR to this branch is https://github.com/scikit-learn/scikit-learn/pull/27663 which implements the base infrastructure for the callbacks and a first callback (progress bars). Subsequent PRs will add more callbacks, mode doc, more tests, adapt estimators to support callbacks, ...

- [ ] Base infrastructure https://github.com/scikit-learn/scikit-learn/pull/27663

**Callbacks**
- [ ] Progress bars https://github.com/scikit-learn/scikit-learn/pull/27663
- [ ] Monitoring
- [ ] EarlyStopping
- [ ] Snapshots
- [ ] Verbose / Logging

**Doc**
- [ ] examples
- [ ] how to write a custom callback

**Adapt estimators**
*full list incoming*
user: jeremiedbb
labels: Meta-issue: General issue associated to an identified list of tasks
issue_27672: state: open
title: Array API cupy fixes
body: I found a few problems when running the existing Array API tests on a machine with CUDA and cupy.

There are a few easy fixes but there is also a problem with `train_test_split` that I did not solve yet.

I don't understand why the existing tests pass with the strict `numpy.array_api`: we should not be able to fancy index an Array using `array[indices, ...]` but instead we should use `array.take(indices, axis=0)`. This needs more investigation but I decided to open an early draft PR to share the existence of the problem with @betatim and others.
user: ogrisel
issue_27666: state: open
title: API rename `Y` to `y` for estimators from the PLS module
body:  I replaced Y for y in fit functions and introduced a deprecation warning in case the user still uses Y.

I have only replaced it on the fit functions. Please let me know if replacing it in the entire module would be better



#### Reference Issues/PRs

Fixes #26945



#### What does this implement/fix? 
The issue mentions that Y should be replaced for y in the sci-kit learn interface. I wasn't sure if it should be for the entire module so I just replaced it on the fit functions. 

#### Any other comments?
No extra comments, this is my first contribution I welcome feedback and criticism.  Thanks!
user: IamjustNick
issue_27663: state: open
title: FEA Callbacks base infrastructure + progress bars
body: Extracted from #22000

This PR implements a smaller portion of #22000, with only the base infrastructure for callbacks and the implementation of a single callback (progress bars). It targets the `callbacks` branch and the goal is to have the full callback implementation done in several smaller PRs merged in this branch before we merge it in main.

This PR proposes significant changes compared to #22000 that should imo improve a lot its change of getting merged :smile:
The main improvement is that it no longer requires writing on the disk, but instead relies on `multiprocessing.Managers` and queues. It simplifies the code a lot.

In #22000 I adapted some estimators to work with the callbacks which I did not include here to keep the PR as light as possible. You can however experiment the callbacks on estimators that I wrote for testing purpose:
```py
from sklearn.callback import ProgressBar
from sklearn.callback.tests._utils import Estimator, MetaEstimator
est = Estimator()
meta_est = MetaEstimator(est, n_jobs=2)
meta_est._set_callbacks(ProgressBar())
meta_est.fit(None, None)
```
You add `sleep` calls in these testing estimators to simulate how it changes when the computations take longer.

The plan is to then have several PRs to implement the other callbacks, adapt a few estimators to work with the callbacks, add some documentation and examples, add more tests...
user: jeremiedbb
issue_27662: state: open
title: PyPy tests timeouts / memory usage investigation
body: I had a closer look at the PyPy tests which have been timing out for a while, here is the result of my investigation. This may also help in the future to have a central issue for this rather than the discussion being split in different automatically created issues in this repo and scikit-learn-feedstock.

The PyPy tests locally needs ~11GB on my machine whereas it is 1.2GB with CPython. I ran them without using pytest-xdist to simplify things.

**PyPy**
![pypy](https://github.com/scikit-learn/scikit-learn/assets/1680079/ffb66c87-92ef-446d-bb2b-4b76a423b915)

**CPython**
![cpython](https://github.com/scikit-learn/scikit-learn/assets/1680079/f3bcca85-0177-428a-9a74-b86009b5c7c8)

It seems like one of the where the memory usage grows with time is the linear_model tests (needs 3.4GB with PyPy and 200MB with CPython locally).

**PyPy**
![linear_model_pypy](https://github.com/scikit-learn/scikit-learn/assets/1680079/ea82ab40-1611-46cc-99dc-baf0d7f52f49)

**CPython**
![linear_model_python](https://github.com/scikit-learn/scikit-learn/assets/1680079/64e6151c-3435-4ae3-8b2b-b524c9f4b32c)

I manage to reproduce the issue (memory growing way more than on CPython) with the following snippet, where one of our Cython loss functions is called many times in a tight loop:

```py
import psutil
import gc
from functools import partial
import platform

import numpy as np

from sklearn._loss.loss import HalfGammaLoss

IS_PYPY = platform.python_implementation() == "PyPy"

def func(data):
    loss = HalfGammaLoss()
    for i in range(10_000):
        loss(data, data)


def main():
    for i in range(101):
        if i % 10 == 0:
            memory_usage = psutil.Process().memory_info().rss / 1e9
            message = f'{i} psutil: {memory_usage:.3f}GB'
            if IS_PYPY:
                pypy_memory_usage = gc.get_stats().memory_allocated_sum
                message += f' , pypy total allocated: {pypy_memory_usage}'
            print(message)

        n_samples, n_features = 4, 12
        rng = np.random.RandomState(0)
        raw_prediction = rng.uniform(low=-3, high=3, size=n_samples)
        func(raw_prediction)

    if IS_PYPY:
        print(gc.get_stats())

if __name__ == '__main__':
    main()
```

Output shows that the memory usage grows with time and that it is about 10 times the total memory allocated reported by PyPy:
```
0 psutil: 0.175GB , pypy total allocated: 69.2MB
10 psutil: 0.537GB , pypy total allocated: 113.9MB
20 psutil: 0.888GB , pypy total allocated: 157.4MB
30 psutil: 1.257GB , pypy total allocated: 201.4MB
40 psutil: 1.595GB , pypy total allocated: 247.4MB
50 psutil: 1.931GB , pypy total allocated: 291.9MB
60 psutil: 2.325GB , pypy total allocated: 330.4MB
70 psutil: 2.659GB , pypy total allocated: 374.4MB
80 psutil: 3.003GB , pypy total allocated: 423.9MB
90 psutil: 3.327GB , pypy total allocated: 460.9MB
100 psutil: 3.663GB , pypy total allocated: 508.4MB
```

Not entirely sure whether this is a red herring, there is an issue with our Cython loss implementation, or this is an issue at the interaction between Cython and PyPy. Maybe @mattip has some insights about this?

On CPython the memory usage is stable around 100MB
```
0 psutil: 0.097GB
10 psutil: 0.098GB
20 psutil: 0.098GB
30 psutil: 0.098GB
40 psutil: 0.098GB
50 psutil: 0.098GB
60 psutil: 0.098GB
70 psutil: 0.098GB
80 psutil: 0.098GB
90 psutil: 0.098GB
100 psutil: 0.098GB
```

<details><summary>My environment for good measure</summary>
<p>

```
 mamba list               
# packages in environment at /home/lesteve/micromamba/envs/pypy:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
cython                    3.0.3            py39hc10206b_0    conda-forge
exceptiongroup            1.1.3              pyhd8ed1ab_0    conda-forge
execnet                   2.0.2              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
gdbm                      1.18                 h0a1914f_2    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
libblas                   3.9.0           19_linux64_openblas    conda-forge
libcblas                  3.9.0           19_linux64_openblas    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_2    conda-forge
libgfortran-ng            13.2.0               h69a702a_2    conda-forge
libgfortran5              13.2.0               ha4646dd_2    conda-forge
libgomp                   13.2.0               h807b86a_2    conda-forge
liblapack                 3.9.0           19_linux64_openblas    conda-forge
libopenblas               0.3.24          pthreads_h413a1c8_0    conda-forge
libsqlite                 3.43.2               h2797004_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_2    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
numpy                     1.26.0           py39h6dedee3_0    conda-forge
openssl                   3.1.4                hd590300_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pip                       23.3               pyhd8ed1ab_0    conda-forge
pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
psutil                    5.9.5            py39hf860d4a_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pypy                      7.3.13                 0_pypy39    conda-forge
pypy3.9                   7.3.13               h9557127_0    conda-forge
pytest                    7.4.2              pyhd8ed1ab_0    conda-forge
pytest-repeat             0.9.2              pyhd8ed1ab_0    conda-forge
pytest-xdist              3.3.1              pyhd8ed1ab_0    conda-forge
python                    3.9.18                0_73_pypy    conda-forge
python_abi                3.9               4_pypy39_pp73    conda-forge
readline                  8.2                  h8228510_1    conda-forge
scipy                     1.11.3           py39h6dedee3_1    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
sqlite                    3.43.2               h2c6b66d_0    conda-forge
threadpoolctl             3.2.0              pyha21a80b_0    conda-forge
tk                        8.6.13               h2797004_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libx11               1.8.7                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
```


</p>
</details> 

user: lesteve
issue_27657: state: open
title: API improve the remainder index dtype to be consistent with transformers
body: 



<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
closes #27533 

#### What does this implement/fix? Explain your changes.

As described in the issue #27533 , this modifies the format of the columns of the last item of the `ColumnTransformer`'s `transformers_` attribute, ie of the item that corresponds to the "remainder". They used to always be indices (integers), now they match the format that was used for the `transformers` parameter, if it was consistent across all transformers:

- if all columns in inputs are provided as column names, so are remainder columns
- if all columns in inputs are provided as boolean masks, so are remainder columns
- otherwise remainder columns are int indices (as before)


This is controlled by the `force_int_remainder_cols` parameter (better name suggestions welcome :)  ) : when it is True the old behavior is kept and a FutureWarning is emitted; when it is False the new behavior is applied


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: jeromedockes
issue_27655: state: open
title: `sklearn.cluster.AgglomerativeClustering`: allow `'ward'` linkage and `'precomputed'` metric.
body: Hi,

I'm trying to run `AgglomerativeClustering` with precomputed (Euclidean) distance matrices. However, I can't get it to work with `linkage='ward'` and `metric='precomputed'` due to this `ValueError`:

https://github.com/scikit-learn/scikit-learn/blob/1f1329f7ecb001eda2ff8e6d6a68bc2054c4962f/sklearn/cluster/_agglomerative.py#L1029-L1033

Would you consider a PR making this into a warning when `metric='precomputed'`? This would allow passing precomputed distance matrices to the Ward method.

Thank you,
Vini
user: vinisalazar
issue_27654: state: open
title: inverse_transform Xt argument consistency 
body: ### Describe the issue linked to the documentation

Some of the inverse_transform methods take `Xt` as an argument whereas others take `X`. Is there are reason for the differences in the names?

Noting the cases here: https://github.com/search?q=repo%3Ascikit-learn%2Fscikit-learn%20%22def%20inverse_transform%22&type=code

### Suggest a potential alternative/fix

Stick to `Xt` in all cases
user: wd60622
issue_27652: state: open
title: Add individual penalization to precision matrix in graphical_lasso.py
body: ### Describe the workflow you want to enable

Friedman et al. (2008) describe the coordinate descent procedure used for the graphical lasso.
In the paper, there is a REMARK 2.1, which states that the objective function to be optimized can be modified to allow for a matrix of penalty values, rather than a scalar value. 

### Describe your proposed solution

This has been implemented [here](https://github.com/m-barylli/scikit-learn-graphical-lasso-edit/commit/05ed1c63dd2f00bbcffbb59db18833d6748dbd54).
However, linear_model._cd_fast.pyx still has to be updated to allow for vectorized alpha input in enet_coordinate_descent_gram

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This change is motivated to allow for prior incorporation into the inference procedure. When strong priors for edges are available, this can affect the strength of the corresponding edges' penalization. 
user: m-barylli
labels: Needs Decision: Requires decision
issue_27636: state: open
title: ENH Add option to disable subsampling to preprocessing.QuantileTransformer
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Closes #27373 

#### What does this implement/fix? Explain your changes.

The current behaviour of the `preprocessing.QuantileTransformer` is to disable subsampling when `subsampling > n_samples` but this is not documented.
This PR adds a documented option (`subsample=None`) to disable subsampling as discussed with @lorentzenchr.

#### Any other comments?

Arguably the default should be changed to `None` in the future as subsampling introduces measurable bias, see #27373 for details.
user: RUrlus
labels: Waiting for Second Reviewer: First reviewer is done, need a second one!
issue_27631: state: open
title: DOC Add dropdowns to User Guide section 3.2, "Tuning the hyper-parameters of an estimator"
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

part of #26617 

#### What does this implement/fix? Explain your changes.

as described in #26617, shorten this section of the user guide by folding some parts in `<details>`

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: jeromedockes
issue_27630: state: open
title: Monotonic trees missing values
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Follow up on #13649 to allow monotonic constraints on tree-based models when missing values are present (introduced in #23595 and #26391).

#### What does this implement/fix? Explain your changes.

Update on tests and monotonic constraints checks to make them work with missing values.

#### Any other comments?

For trees built with missing values, I had to work around:
- unexpected middle values (still not able to understand them, opening a debug branch to inspect trees better)
- the clipping mechanism modifying the nodes upper and lower bounds and making bound checking fail in `assert_nd_reg_tree_children_monotonic_bounded`.

Current implementation works, but I'm not satisfied with having to deal with middle values outside of the bounds. Will investigate further.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: samronsin
issue_27629: state: open
title: Please provide option to set unknown_values during test time to same as encoded min_frequency  in OrdinalEncoder(Infrequent categories)
body: ### Describe the workflow you want to enable

It seems that OneHotEncoder has a parameter for setting` handle_unknown='infrequent_if_exist'` but the same is missing in OrdinalEncoder . Currently `unknown_value` and the value encoded by setting the parameter `min_frequency` seems to be different. There is always workaround to figure out the encoded value on `min_frequency` and pass the same to `unknown_values` but I think having something similar to OneHotEncoder's parameter `handle_unknown='infrequent_if_exist'` seems intuitive as we would want to treat unseen values as infrequent ones. Not sure if this feature already exists and I'm missing it somehow. 

### Describe your proposed solution

Implement parameter option similar to OneHotEncoder's parameter `handle_unknown='infrequent_if_exist'`  where unknown (unseen values during training) get similar encoding as happened for infrequent_categories during training. 

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: abhishek0093
labels: Hard: Hard level of difficulty
issue_27620: state: open
title: sklearn PCA rotates a single vector
body: ### Describe the bug

The issue we recently discovered is that sklearn PCA rotates the input when only a single variable is fed into the model.  

I am aware there are infinite rotations when there is a single vector fed into the model, however, the output from PCA should intuitively make sense. So I suggest hard coding this scenario (which I guess already done in R)


### Steps/Code to Reproduce
```python
import numpy as np
from sklearn.decomposition import PCA

x = np.array([1,2,3,4,5,6,7,8,9,10.]).reshape(-1,1).astype('float64')

p = PCA().fit_transform(x)
p
```
### output:
```
array([[ 4.5],
       [ 3.5],
       [ 2.5],
       [ 1.5],
       [ 0.5],
       [-0.5],
       [-1.5],
       [-2.5],
       [-3.5],
       [-4.5]])
```

as we see here the value 4.5 is corresponding to the smallest value in the set (1) and the value -4.5 is corresponding to the highest value in the input set (10).

### Expected Results:
```
array([[-4.5],
       [-3.5],
       [-2.5],
       [-1.5],
       [-0.5],
       [ 0.5],
       [ 1.5],
       [ 2.5],
       [ 3.5],
       [ 4.5]])
```



### Steps/Code to Reproduce

```
from sklearn.decomposition import PCA

x= np.array([1,2,3,4,5,6,7,8,9,10.]).reshape(-1,1).astype('float64')

p = PCA().fit_transform(x)
p
```

### Expected Results

```
array([[-4.5],
       [-3.5],
       [-2.5],
       [-1.5],
       [-0.5],
       [ 0.5],
       [ 1.5],
       [ 2.5],
       [ 3.5],
       [ 4.5]])
```

### Actual Results

```
array([[ 4.5],
       [ 3.5],
       [ 2.5],
       [ 1.5],
       [ 0.5],
       [-0.5],
       [-1.5],
       [-2.5],
       [-3.5],
       [-4.5]])
```

### Versions

```shell
System:
    python: 3.11.4 | packaged by Anaconda, Inc. | (main, Jul  5 2023, 13:47:18) [MSC v.1916 64 bit (AMD64)]
executable: e:\Users\***\miniconda3\python.exe
   machine: Windows-10-10.0.22621-SP0

Python dependencies:
      sklearn: 1.3.0
          pip: 23.2.1
   setuptools: 68.0.0
        numpy: 1.24.3
        scipy: 1.11.1
       Cython: None
       pandas: 2.0.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       filepath: E:\Users\***\miniconda3\Library\bin\mkl_rt.2.dll
         prefix: mkl_rt
       user_api: blas
   internal_api: mkl
        version: 2023.1-Product
    num_threads: 8
threading_layer: intel

       filepath: E:\Users\***\miniconda3\vcomp140.dll
         prefix: vcomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 16

       filepath: E:\Users\***\miniconda3\Library\bin\libiomp5md.dll
         prefix: libiomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 16
```

user: hamedha2
issue_27593: state: open
title: Deprecate murmurhash3_32
body: `sklearn.utils.murmurhash3_32` is part of our API, but we don't use it anywhere internally.
I propose to deprecate and finally remove it. The standard Python [`hash`](https://docs.python.org/3/library/functions.html#hash) function using SipHash per [PEP0456](https://peps.python.org/pep-0456/) might serve as a good replacement depending on the use case.
user: lorentzenchr
labels: Needs Decision - Backward Compatibility: Requires decision regarding backward compatibility
issue_27592: state: open
title: Using tqdm or progress bars while downloading datasets using `urlretreve`
body: ### Describe the workflow you want to enable

https://github.com/scikit-learn/scikit-learn/blob/4ca01961969a0c9e1c7c48410e0976bb04a92703/sklearn/datasets/_base.py#L1368-L1399

When we fetch remote data using the function `_fetch_remote`, we cannot verify that we are downloading it normally because there is no interaction action.

Depending on the download environment, it may take a long time, and users may think that there is an error in the program if they do not see it.

### Describe your proposed solution

I suggest that make an option to display the progress bar that we download using tqdm or other methods.

We can simply implement using [urlretreve reporthook parameter](https://docs.python.org/3/library/urllib.request.html#urllib.request.urlretrieve), Here are some example.

https://gist.github.com/leimao/37ff6e990b3226c2c9670a2cd1e4a6f5

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I've also suggest that adding more options: timeout, retrying.
user: mirusu400
issue_27588: state: open
title: ENH Add Array API compatibility to `Binarizer`
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Towards https://github.com/scikit-learn/scikit-learn/issues/26024
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Tialo
issue_27572: state: open
title: ENH Introduces `mst_algorithm` keyword for HDBSCAN, alongside two new Boruvka MST algorithms
body: #### Reference Issues/PRs
Towards https://github.com/scikit-learn/scikit-learn/issues/26801

#### What does this implement/fix? Explain your changes.
- Introduces `mst_algorithm`
- Adds Boruvka algorithm (both exact/inexact)
- Streamlines algorithm selection logic for "auto" options
- Improves tests to account for new MST algorithms
- Provides a `"warn"` option for `mst_algorithm` as a backwards-compatible default which will provide a `FutureWarning` to users, encouraging them to opt-in to using `mst_algorithm="auto"`
- Includes a deprecation for `mst_algorithm="warn"`

#### Any other comments?
Apologies for the gross commit log, the vast majority are a symptom of me keeping this branch open in parallel with mainstream HDBSCAN efforts in order to prevent it from getting "too out of sync". Not sure how to rectify history well here.

Benchmarks coming soon.
user: Micky774
issue_27568: state: open
title: Instable test test_logistic_regressioncv_class_weights due to ConvergenceWarning
body: When merging #26271, we added a lower tolerance in the following line:

https://github.com/scikit-learn/scikit-learn/pull/26721/files#diff-84dc33502e99175d160109f0ee16007297dc398d872a63c977b787287ec4ed00R752

However, it fails time to time on Ubuntu Atlas and the 32 bits docker CIs.
We also have a tolerance set to `1e-18` for the SAG and SAGA.

Should we increase the tolerance?

@lorentzenchr 
user: glemaitre
issue_27566: state: open
title: FEA Add metadata routing to GraphicalLassoCV
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Towards #22893

#### What does this implement/fix? Explain your changes.
- Adds metadata routing to GraphicalLassoCV and routes the corresponding metadata to the underlying cv splitter and the cross_val_score function.

#### Any other comments?
CC: @adrinjalali @glemaitre 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: OmarManzoor
issue_27560: state: open
title: FEA Add metadata routing to RidgeCV and RidgeClassifierCV
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Towards: #22893

#### What does this implement/fix? Explain your changes.
- Adds metadata routing to RidgeCV and RidgeClassifierCV

#### Any other comments?
CC: @adrinjalali @glemaitre 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: OmarManzoor
issue_27559: state: open
title: Correctly document linked libraries
body: ### Describe the issue linked to the documentation

When downloading the current wheel for `scikit-learn==1.3.1`, the metadata tell me that the package is subject to the terms of BSD-3-Clause. Unfortunately, this only applies to the package itself. Skimming through the distributed files, there are at least two additional cases:

* External code snippets under licenses like MIT, Apache-2.0 and Python-2.0
* Binary modules like `libgomp-a34b3233.so.1.0.0`, subject to GPL-3.0-or-later WITH Runtime exception: https://github.com/gcc-mirror/gcc/blob/master/libgomp/libgomp.h


### Suggest a potential alternative/fix

It would be great if a full list of external modules shipped within *scikit-learn* wheels and their copyright information would be provided to detect possible license conflicts early.
user: stefan6419846
issue_27535: state: open
title: Use float32_t for tree.threshold
body: The features `X` in our standard decision trees are float32, so it would make sense for the threshold of features to also be float32, see https://github.com/scikit-learn/scikit-learn/blob/8ae5f186986667bc3042a36f5d23e352acc40154/sklearn/tree/_tree.pxd#L31

Note that the Cython trees are expose in our trees, e.g. [`DecisionTreeRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html#sklearn.tree.DecisionTreeRegressor`) attribute `tree_`.
user: lorentzenchr
labels: Needs Decision - Backward Compatibility: Requires decision regarding backward compatibility
issue_27533: state: open
title: Better inference of the columns remainder dtype in `transformers_` from `ColumnTransformer`
body: A typical use case is to fit a `ColumnTransfomrer` on a pandas dataframe such as:

```python
# %%
from sklearn.datasets import load_iris

df, y = load_iris(return_X_y=True, as_frame=True)

# %%
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer

cols = ["sepal length (cm)", "sepal width (cm)"]
ct = ColumnTransformer([("scale", StandardScaler(), cols)], remainder="passthrough")
ct.fit_transform(df)
```

While investigating the `transformers_`, the columns exposed for `remainder` is a bit weird:

```python
ct.transformers_
```

```
[('scale', StandardScaler(), ['sepal length (cm)', 'sepal width (cm)']),
 ('remainder', 'passthrough', [2, 3])]
```

In terms of UX, I assume that one would expect to get the feature names in the remainder instead of the feature indices. However, our API to pass the information of the columns in scikit-learn is pretty flexible: we can have missed type of slice, array of int, etc.

I would suggest that when the columns passed by the user are of a single type, then we make sure that the column type of the `remainder` is of the same type: if only indices are passed, we show indices, if names are passed, we show names, and if boolean are passed, we show boolean.

With mixed types, we cannot really decide and we can keep the current behaviour. On a UX perspective, I assume this is better (together with the change in #27204).

Any thoughts @lorentzenchr @adrinjalali @ogrisel @betatim @jeremiedbb 

user: glemaitre
issue_27530: state: open
title: SimpleImputer strategy "most_frequent" returning ValueError: could not convert string to float when imputing strings
body: I want to use sklearn.preprocessing.SimpleImputer, with strategy='most_frequent", in a pipeline to impute missing values in a categorical feature column of a dataframe. However, when I execute the pipeline, it raises the following error:
ValueError: could not convert string to float:<35'
I am under the impression that the 'most_frequent' setting for strategy can be used on categorical data. So, I do not understand why this tries to convert a string to a float? It seems to contradict the SimpleImputer documentation.

Here's the code which raises the error:
```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import FunctionTransformer, StandardScaler, OrdinalEncoder, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.feature_extraction import FeatureHasher
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.cluster import KMeans

#FunctionTransformer applies this function to the entire dataframe
def Funct(X=None):
  return X.apply(lambda x:np.sqrt(x))

class ColDropTransformer(BaseEstimator,TransformerMixin):
  def __init__(self,collist=None):
    self.collist = collist

  def fit(self,X,y=None):
    return self

  def transform(self,X,y=None):
    return X.drop(self.collist,axis=1)

class StringTransformer(BaseEstimator,TransformerMixin):
  def __init__(self,col=None):
    self.col = col

  def fit(self,X,y=None):
    return self

  def transform(self,X,y=None):
    print(f'NaNs:{X.isna().sum()}##########')
    print(f'X to do string split:{X}')
    print(f'ABOUT TO SPLIT STRINGS')
    X[self.col] = X[self.col].str.split(';')
    print(f'X after split:{X}')
    return X

class Debugger(BaseEstimator, TransformerMixin):
  def fit(self,X,y=None,**fit_params):
    return self

  def transform(self,X):
    print(pd.DataFrame(X).head())

data = pd.read_csv('stackoverflow_full.csv')

target_feature = ['Employed']
X = data.drop(target_feature,axis=1)
y = data[target_feature].copy()
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2,random_state=42)
X_train, X_eval, y_train, y_eval = train_test_split(X_train,y_train,test_size=0.01,random_state=42)


#The features to process
debug_cols = list(data.columns)
debug_cols = debug_cols.remove('Unnamed: 0')
drop_cols = ['Unnamed: 0']
numeric_cols = ['Employment','YearsCode','YearsCodePro','PreviousSalary','ComputerSkills']
cat_cols = ['Age','Accessibility','EdLevel','Gender','MentalHealth','MainBranch','Country','HaveWorkedWith']
onehot_cols = ['Age','Accessibility','Employment','Gender','MentalHealth','MainBranch']
ordinal_cols = ['EdLevel']
hash_cols = ['Country','HaveWorkedWith']
string_split_cols = ['HaveWorkedWith']

#The transformers to be used
funct_tfmr = FunctionTransformer()
str_tfmr = StringTransformer()
hash_tfmr = FeatureHasher(n_features=1000,input_type='string')
num_impute_tfmr = SimpleImputer()
cat_impute_tfmr = SimpleImputer(strategy="most_frequent")
onehot_tfmr = OneHotEncoder(handle_unknown='ignore')
ordinal_tfmr = OrdinalEncoder(handle_unknown='use_encoded_value',unknown_value=-1)
coldrop_tfmr = ColDropTransformer(drop_cols)
scale_tfmr = StandardScaler()

transformer = ColumnTransformer(transformers=[('drop col',coldrop_tfmr,drop_cols),('impute_nums',num_impute_tfmr,numeric_cols),('impute_cats',cat_impute_tfmr,cat_cols),('sqrt',funct_tfmr,numeric_cols),('num_scale',scale_tfmr,numeric_cols),('onehot',onehot_tfmr,onehot_cols),('ordinal',ordinal_tfmr,ordinal_cols),('str_split',str_tfmr,string_split_cols),('hashing',hash_tfmr,hash_cols)],verbose=True)

pipe = Pipeline(steps=[('transform',transformer),('cluster',KMeans(n_clusters=2))],verbose=True)

pipe.fit_transform(X_train,y_train)
```
The 'Age' column has 2 values ('<35' , '>35') and this is the first, but not only, categorical feature column causing this error to be raised.

Any help will be appreciated as I am lost on how to fix this
[stackoverflow_full.csv](https://github.com/scikit-learn/scikit-learn/files/12803457/stackoverflow_full.csv)

user: JacquesSA
issue_27528: state: open
title: Extra plots in partial dependence plots
body: ### Describe the workflow you want to enable

As discussed in #19410, there has been interest in including additional visualizations along with the partial dependence visualizations. Extra plots would aid in the interpretation of partial dependence plots. It would be low overhead for the user to specify "hist" as an argument and have the feature distribution plotted in the same figure as the partial dependence plot.  This issue only addresses the suggestion to improve partial dependence plots, not ICE plots.

### Describe your proposed solution

#27388 introduces three new parameters to the `from_estimator` method in the `PartialDependenceDisplay` class:

* `extra_plots`: A string or list of strings specifying what type of extra plot to include in the partial dependence display.
* `extra_plots_kw`: A dictionary where the keys should match the plot type specified in `extra_plots` and the values are kwarg dictionaries for each plot type.
* `y`: Only used in one-way partial dependence plots when the extra plot is a scatter plot. 

Their defaults are all `None`, which does not affect the current PDP display behavior.
The following image can be generated by simply adding: `extra_plots=["boxplot", "hist"]` to the `from_estimator` call.

![image](https://github.com/scikit-learn/scikit-learn/assets/9151717/03483a24-a238-4594-be7f-49ae5437eac3)


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: arvkevi
issue_27526: state: open
title: ENH add verbosity to newton-cg solver
body: #### Reference Issues/PRs
This PR is meant to be merged after #26721.

#### What does this implement/fix? Explain your changes.
This PR adds verbosity to `_newton_cg` solver in our private `sklearn.utils.optimize` module. It is used, e.g., in `LogisticRegression`.

#### Any other comments?
user: lorentzenchr
issue_27522: state: open
title: Add new estimator checks for sparse arrays to ensure compatibility for third-party libraries
body: In #27090, we make changes in our tests to check that our estimators are compatible with sparse arrays. However, it does not intend to write common tests through new checks available in `estimator_checks.py`.

We should implement new checks that are testing the same as the sparse matrix tests to ensure that third-party library will be compatible with such arrays. However, we want to be extra-nice: we should also create a new tag, e.g. `X_types: {..., "sparse_arrays", "sparse matrix"}` and run the tests only when the types is explicitly supported.

The documentation states that we are not currently using the `"sparse"` key, so we should be able to introduce new keys without any regression.
user: glemaitre
labels: module:test-suite: everything related to our tests
issue_27519: state: open
title: ENH add verbose option to SequentialFeatureSelector
body: add verbose option to SequentialFeatureSelector
user: jimmy927
issue_27516: state: open
title: ENH Implement monotonicity constraints for `GradientBoosting*`
body: #### Reference Issues/PRs

Closes #27305.

#### What does this implement/fix? Explain your changes.

This PR implements monotonicity constraints for `GradientBoostingClassifier` and `GradientBoostingRegressor`. This is dropped from #13649.

#### Any other comments?

For your reference: [Greedy Function Approximation, Friedman](https://jerryfriedman.su.domains/ftp/trebst.pdf). There were discussions around whether line search should be performed when using monotonic constraints, see https://github.com/scikit-learn/scikit-learn/pull/13649#discussion_r973630373. I actually did not fully understand this so it would be nice if someone can explain in more details. By the way, `test_monotonic_constraints_classifications` in `sklearn/tree/tests/test_monotonic_tree.py` would fail if line search is performed.

Speaking of tests, I'm also a bit confused where they should be placed. It seems that we should have similar (if not the same) tests as `sklearn/tree/tests/test_monotonic_tree.py` so I currently only  extended the parametrizations to include `GradientBoostingClassifier` and `GradientBoostingRegressor`.  Still, it's a bit strange to test one module under another. Please correct me if this is wrong.

@lorentzenchr Would you want to take a look? I'm not sure if this is what the target issue desired.
user: Charlie-XIAO
issue_27514: state: open
title: Model Persistence doc page could provide clearer actionable recommendations
body: ### Describe the issue linked to the documentation

The [Model Persistence page](https://github.com/scikit-learn/scikit-learn/blob/main/doc/model_persistence.rst) currently discusses many options (pickling, `skops`, ONNX and PMML, but it does it sequentially (first discusses one, then the other, and so on) without a clear narration. I think most people would not know what they should do after reading that page.

I understand these pages are not supposed to be tutorials or blogposts, but only high-level directions on the path to take, but still I think we could have a better page.

Doing MLOps consulting work, I've found most Data Scientists have big misconceptions about what they should do in order to correctly and safely save and then load scikit-learn models. In particular, most Data Scientists I've met think pinning the scikit-learn version is enough to ensure compatibility. The documentation is clear in this regard, but maybe dances around too much between options to make the point something people will remember.

### Suggest a potential alternative/fix

I think there's a couple of changes that could be done in order to improve the narrative and make sure the basic points come across to everyone, while still discussing all available options as discussed now.

1. List all alternatives near the beginning of the document.
2. _Start_ by discussing the fact that you need to pin all transitive scikit-learn dependencies to be able to safely load a pickle, which is the format most people will use, and probably gets you 80% of the benefit you will get from more complicated recommendations.
2. Do more to compare the presented alternatives (what does skops lack that would make one want to use ONNX?)
3. Summarize the most important points at the end of the document.

I can provide a PR if you agree these changes are desirable. 
user: IanTayler
issue_27511: state: open
title: TST Extend tests for `scipy.sparse.*array` in `sklearn/svm/tests/test_sparse.py`
body: Refer Issue #27090 

currently a draft, would love to take any help.
user: work-mohit
issue_27508: state: open
title: Mention that DBSCAN might modify precomputed sparse distance matrix
body: ### Describe the issue linked to the documentation

[DBSCAN](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html) provides a parameter called `metric` which can be assigned the value `precomputed` so that a precomputed distance matrix can be passed to the `fit` method. 

> If metric is “precomputed”, X is assumed to be a distance matrix and must be square. X may be a [sparse graph](https://scikit-learn.org/stable/glossary.html#term-sparse-graph), in which case only “nonzero” elements may be considered neighbors for DBSCAN.

When a sparse matrix is passed to DBSCAN, it is modified [in-place.](https://github.com/scikit-learn/scikit-learn/blob/2d8e03f4d5b3c466fb4542360bcef742a3a4e0a1/sklearn/cluster/_dbscan.py#L384-L389). If the sparse matrix already had diagonal elements present, this leads to no change in the matrix. However, if the diagonal elements are not present, then extra elements are added to the sparse matrix. In the documentation, this should be mentioned explicitly. 

### Suggest a potential alternative/fix

For the `metric` param - 

> If metric is “precomputed”, X is assumed to be a distance matrix and must be square. X may be a [sparse graph](https://scikit-learn.org/stable/glossary.html#term-sparse-graph), in which case only “nonzero” elements may be considered neighbors for DBSCAN. **Note that DBSCAN modifies the sparse matrix in-place by setting the diagonal of the sparse matrix.**
user: tataganesh
issue_27506: state: open
title: Test failure in i686 with version 1.3.1
body: ### Describe the bug

During the build of scikit-learn for Fedora Linux, I'm obtaining an error runing the tests in i686. The test that fails is:

`sklearn/tree/tests/test_export.py::test_graphviz_toy`

### Steps/Code to Reproduce

In a i686 machine

```
pytest sklearn/tree/tests/test_export.py
```

### Expected Results

Test passes

### Actual Results

```
sklearn/tree/tests/test_export.py::test_graphviz_toy FAILED              [ 93%]
=================================== FAILURES ===================================
______________________________ test_graphviz_toy _______________________________
    def test_graphviz_toy():
        # Check correctness of export_graphviz
        clf = DecisionTreeClassifier(
            max_depth=3, min_samples_split=2, criterion="gini", random_state=2
        )
        clf.fit(X, y)
    
        # Test export code
        contents1 = export_graphviz(clf, out_file=None)
        contents2 = (
            "digraph Tree {\n"
            'node [shape=box, fontname="helvetica"] ;\n'
            'edge [fontname="helvetica"] ;\n'
            '0 [label="x[0] <= 0.0\\ngini = 0.5\\nsamples = 6\\n'
            'value = [3, 3]"] ;\n'
            '1 [label="gini = 0.0\\nsamples = 3\\nvalue = [3, 0]"] ;\n'
            "0 -> 1 [labeldistance=2.5, labelangle=45, "
            'headlabel="True"] ;\n'
            '2 [label="gini = 0.0\\nsamples = 3\\nvalue = [0, 3]"] ;\n'
            "0 -> 2 [labeldistance=2.5, labelangle=-45, "
            'headlabel="False"] ;\n'
            "}"
        )
    
        assert contents1 == contents2
    
        # Test plot_options
        contents1 = export_graphviz(
            clf,
            filled=True,
            impurity=False,
            proportion=True,
            special_characters=True,
            rounded=True,
            out_file=None,
            fontname="sans",
        )
        contents2 = (
            "digraph Tree {\n"
            'node [shape=box, style="filled, rounded", color="black", '
            'fontname="sans"] ;\n'
            'edge [fontname="sans"] ;\n'
            "0 [label=<x<SUB>0</SUB> &le; 0.0<br/>samples = 100.0%<br/>"
            'value = [0.5, 0.5]>, fillcolor="#ffffff"] ;\n'
            "1 [label=<samples = 50.0%<br/>value = [1.0, 0.0]>, "
            'fillcolor="#e58139"] ;\n'
            "0 -> 1 [labeldistance=2.5, labelangle=45, "
            'headlabel="True"] ;\n'
            "2 [label=<samples = 50.0%<br/>value = [0.0, 1.0]>, "
            'fillcolor="#399de5"] ;\n'
            "0 -> 2 [labeldistance=2.5, labelangle=-45, "
            'headlabel="False"] ;\n'
            "}"
        )
    
        assert contents1 == contents2
    
        # Test max_depth
        contents1 = export_graphviz(clf, max_depth=0, class_names=True, out_file=None)
        contents2 = (
            "digraph Tree {\n"
            'node [shape=box, fontname="helvetica"] ;\n'
            'edge [fontname="helvetica"] ;\n'
            '0 [label="x[0] <= 0.0\\ngini = 0.5\\nsamples = 6\\n'
            'value = [3, 3]\\nclass = y[0]"] ;\n'
            '1 [label="(...)"] ;\n'
            "0 -> 1 ;\n"
            '2 [label="(...)"] ;\n'
            "0 -> 2 ;\n"
            "}"
        )
    
        assert contents1 == contents2
    
        # Test max_depth with plot_options
        contents1 = export_graphviz(
            clf, max_depth=0, filled=True, out_file=None, node_ids=True
        )
        contents2 = (
            "digraph Tree {\n"
            'node [shape=box, style="filled", color="black", '
            'fontname="helvetica"] ;\n'
            'edge [fontname="helvetica"] ;\n'
            '0 [label="node #0\\nx[0] <= 0.0\\ngini = 0.5\\n'
            'samples = 6\\nvalue = [3, 3]", fillcolor="#ffffff"] ;\n'
            '1 [label="(...)", fillcolor="#C0C0C0"] ;\n'
            "0 -> 1 ;\n"
            '2 [label="(...)", fillcolor="#C0C0C0"] ;\n'
            "0 -> 2 ;\n"
            "}"
        )
    
        assert contents1 == contents2
    
        # Test multi-output with weighted samples
        clf = DecisionTreeClassifier(
            max_depth=2, min_samples_split=2, criterion="gini", random_state=2
        )
        clf = clf.fit(X, y2, sample_weight=w)
    
>       contents1 = export_graphviz(clf, filled=True, impurity=False, out_file=None)
sklearn/tree/tests/test_export.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn/utils/_param_validation.py:211: in wrapper
    return func(*args, **kwargs)
sklearn/tree/_export.py:905: in export_graphviz
    exporter.export(decision_tree)
sklearn/tree/_export.py:465: in export
    self.recurse(decision_tree.tree_, 0, criterion=decision_tree.criterion)
sklearn/tree/_export.py:528: in recurse
    ', fillcolor="%s"' % self.get_fill_color(tree, node_id)
sklearn/tree/_export.py:285: in get_fill_color
    return self.get_color(node_val)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <sklearn.tree._export._DOTTreeExporter object at 0xc5080408>
value = -0.4691358024691358
    def get_color(self, value):
        # Find the appropriate color & intensity for a node
        if self.colors["bounds"] is None:
            # Classification tree
            color = list(self.colors["rgb"][np.argmax(value)])
            sorted_values = sorted(value, reverse=True)
            if len(sorted_values) == 1:
                alpha = 0.0
            else:
                alpha = (sorted_values[0] - sorted_values[1]) / (1 - sorted_values[1])
        else:
            # Regression tree or multi-output
            color = list(self.colors["rgb"][0])
            alpha = (value - self.colors["bounds"][0]) / (
                self.colors["bounds"][1] - self.colors["bounds"][0]
            )
        # compute the color as alpha against white
>       color = [int(round(alpha * c + (1 - alpha) * 255, 0)) for c in color]
E       ValueError: cannot convert float NaN to integer
```

### Versions

```shell
Version is 1.3.1
```

user: sergiopasra
issue_27491: state: open
title: ENH new svd_solver='covariance_eigh' for PCA
body: Implements the new solver proposed in #27483.

I also took the opportunity of this PR to strengthen the solver equivalence tests.

## Benchmark

### Laptop CPU with numpy on macOS (with Apple Accelerate BLAS)

```python
>>> import numpy as np
>>> from sklearn.datasets import make_low_rank_matrix
>>> X = make_low_rank_matrix(n_samples=int(1e6), n_features=100).astype(np.float32)
>>> from sklearn.decomposition import PCA
>>> %time PCA(n_components=5, svd_solver="randomized").fit_transform(X)[-1]
CPU times: user 1.67 s, sys: 186 ms, total: 1.85 s
Wall time: 1.1 s
array([ 0.00055511, -0.00047962,  0.0009717 , -0.00128793, -0.00048758],
      dtype=float32
>>> %time PCA(n_components=5, svd_solver="full").fit_transform(X)[-1]
CPU times: user 11.8 s, sys: 298 ms, total: 12.1 s
Wall time: 3.56 s
array([ 0.00055506, -0.00047963,  0.00097174, -0.00128779, -0.00048765],
      dtype=float32)
>>> %time PCA(n_components=5, svd_solver="covariance_eigh").fit_transform(X)[-1]
CPU times: user 121 ms, sys: 3.3 ms, total: 124 ms
Wall time: 123 ms
array([ 0.00055505, -0.00047963,  0.00097174, -0.0012878 , -0.00048764],
      dtype=float32)
```

so around a 10x speed-up.

Switching to OpenBLAS makes it 2x or so slower than Apple Accelerate but still very good.

Note: to switch the BLAS implementation with conda-forge, there is no need to rebuild scikit-learn or scipy. Just the following commands:

```
mamba install "libblas=*=*accelerate"
```
or
```
mamba install "libblas=*=*openblas"
```

### PyTorch via Array API with MPS device on Apple M1

```python
>>> import numpy as np
>>> from sklearn.datasets import make_low_rank_matrix
>>> X = make_low_rank_matrix(n_samples=int(1e6), n_features=100).astype(np.float32)
>>> import torch
>>> X_torch_mps = torch.tensor(X).to("mps")
>>> import sklearn
>>> sklearn.set_config(array_api_dispatch=True)
>>> from sklearn.decomposition import PCA
>>> %time PCA(n_components=5, svd_solver="randomized", power_iteration_normalizer="QR").fit_transform(X_torch_mps)[-1]
/Users/ogrisel/code/scikit-learn/sklearn/utils/extmath.py:319: UserWarning: The operator 'aten::linalg_qr.out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/miniforge3/conda-bld/pytorch-recipe_1690825982320/work/aten/src/ATen/mps/MPSFallback.mm:11.)
  Q, _ = normalizer(A @ Q)
CPU times: user 3.48 s, sys: 474 ms, total: 3.96 s
Wall time: 3.02 s
tensor([ 0.0003, -0.0009,  0.0004, -0.0002, -0.0002], device='mps:0')
>>> %time PCA(n_components=5, svd_solver="full").fit_transform(X_torch_mps)[-1]
CPU times: user 11.4 s, sys: 483 ms, total: 11.9 s
Wall time: 3.83 s
tensor([ 0.0003, -0.0009,  0.0004, -0.0002, -0.0002], device='mps:0'
>>> %time PCA(n_components=5, svd_solver="covariance_eigh").fit_transform(X_torch_mps)[-1]
CPU times: user 11.6 ms, sys: 69.7 ms, total: 81.4 ms
Wall time: 368 ms
tensor([ 0.0003, -0.0009,  0.0004, -0.0002, -0.0002], device='mps:0')
```

As the warning said, the MPS backend for PyTorch still lacks `torch.linalg.*` kernels and fallbacks to the CPU which makes the overall performance a bit slower than just using numpy on CPU. I can also try directly using PyTorch on the CPU (without the MPS backend):

```python
>>> X_torch_cpu = torch.tensor(X)
>>> %time PCA(n_components=5, svd_solver="randomized", power_iteration_normalizer="QR").fit_transform(X_torch_cpu)[-1]
CPU times: user 3.12 s, sys: 692 ms, total: 3.81 s
Wall time: 1.44 s
tensor([ 0.0003, -0.0009,  0.0004, -0.0002, -0.0002])
>>> %time PCA(n_components=5, svd_solver="full").fit_transform(X_torch_cpu)[-1]
CPU times: user 11.3 s, sys: 535 ms, total: 11.8 s
Wall time: 3.23 s
tensor([ 0.0003, -0.0009,  0.0004, -0.0002, -0.0002])
>>> %time PCA(n_components=5, svd_solver="covariance_eigh").fit_transform(X_torch_cpu)[-1]
CPU times: user 198 ms, sys: 18.3 ms, total: 217 ms
Wall time: 112 ms
tensor([ 0.0003, -0.0009,  0.0004, -0.0002, -0.0002])
```

So we get similar results as with numpy on the same host.

### Intel Xeon CPU (44 physical cores from 2018 or so) with numpy on Linux (with MKL BLAS)

I get similar numbers (slightly slower) as on my macOS laptop for the new solver. I inspected the use of threads with htop (on a 10x larger dataset) and I could see that MKL (and OpenBLAS) decide to only use 1 thread for the matrix matrix multiplication (to compute the covariance). This is probably because the matrix is very skinny and concurrent updates by several threads would be detrimental. As a result, using a multi-core machine is not very beneficial for this solver on this kind of data (but it's still more 5x faster than using the randomized solver that actually uses many cores on htop, but not 100% of the time).

Switching to OpenBLAS makes it 2x or so slower than MKL but still very good.

Anyways, the performance (with MKL) seems to be competitive with the [scikit-learn-intelex benchmark results](https://mbatoul.github.io/sklearn_benchmarks/results/github_ci/master/20220322T183227/scikit_learn_intelex_vs_scikit_learn.html#speedup-barplots-pca) we collected a while ago. Note that I cannot reproduce those values with the latest version of scikit-learn-intelex because it seems to use the scikit-learn "randomized" solver instead of their own in the latest version.

### PyTorch via Array API with cuda (NVIDIA V100 GPU)

```python
>>> import numpy as np
>>> from sklearn.datasets import make_low_rank_matrix
>>> X = make_low_rank_matrix(n_samples=int(1e6), n_features=100).astype(np.float32)
>>> import torch
>>> X_torch_cuda = torch.tensor(X).to("cuda")
>>> import sklearn
>>> sklearn.set_config(array_api_dispatch=True)
>>> from sklearn.decomposition import PCA
>>> %time PCA(n_components=5, svd_solver="randomized", power_iteration_normalizer="QR").fit_transform(X_torch_cuda)[-1]
CPU times: user 63.8 ms, sys: 15.5 ms, total: 79.3 ms
Wall time: 78.3 ms
tensor([ 5.4981e-06,  7.2371e-04,  5.3485e-04, -1.3649e-03, -1.4326e-03],
       device='cuda:0')
>>> %time PCA(n_components=5, svd_solver="full").fit_transform(X_torch_cuda)[-1]
CPU times: user 81.4 ms, sys: 19.7 ms, total: 101 ms
Wall time: 99.8 ms
tensor([ 5.5376e-06,  7.2371e-04,  5.3483e-04, -1.3650e-03, -1.4326e-03],
       device='cuda:0')
>>> %time PCA(n_components=5, svd_solver="covariance_eigh").fit_transform(X_torch_cuda)[-1]
CPU times: user 7.67 ms, sys: 362 µs, total: 8.04 ms
Wall time: 7.21 ms
tensor([ 5.4768e-06,  7.2370e-04,  5.3485e-04, -1.3650e-03, -1.4326e-03],
       device='cuda:0')
```

So here again we observe a 10x speed-up with the new solver.

We can compare to NVIDIA's cuML from RAPIDS 23.08 on the same data either provided as CPU allocated numpy array or GPU allocated CuPy:

```python
>>> import numpy as np
>>> from sklearn.datasets import make_low_rank_matrix
>>> X = make_low_rank_matrix(n_samples=int(1e6), n_features=100).astype(np.float32)
>>> from cuml.decomposition import PCA
>>> %time PCA(n_components=5).fit_transform(X)[-1]
CPU times: user 177 ms, sys: 42.5 ms, total: 220 ms
Wall time: 224 ms
array([-0.00171027,  0.00036515, -0.00089102,  0.00108428, -0.00054887],
      dtype=float32)
>>> import cupy
>>> X_cupy = cupy.asarray(X)
>>> %time PCA(n_components=5).fit_transform(X_cupy)[-1]
CPU times: user 25.3 ms, sys: 11.5 ms, total: 36.8 ms
Wall time: 36.8 ms
array([-0.00171027,  0.00036515, -0.00089102,  0.00108428, -0.00054887],
      dtype=float32)
```

So when the data is pre-allocated on the cuda device, our PyTorch + cuda implementation is 5x faster than cuML's!

### PyTorch via Array API on Intel Max GPU

The performance is not good at all, but I suspect that the Intel extension for PyTorch is not ready yet. I opened an issue to track progress upstream:

- https://github.com/intel/intel-extension-for-pytorch/issues/439

## TODO:

- [x] add a changelog entry,
- [x] run some benchmarks,
- [x] check that Array API compliance tests works on cuda, including with the new solver
- [x] compare the new solver to cuML on GPU
- [x] fix remaining CI failures,
- [x] improve the test by also checking consistent output for `transform` and `fit_transform`,
- [x] add a new commit to use the new solver when it makes sense by updating the dimension dependent benchmark scripts (to be discussed next),
- [x] compare the new solver to scikit-learn-intelex on CPU

Note that I had to change the way we use `svd_flip` in this estimator to be able to use get the same signs for the components of the new solver. I think it's better for scikit-learn to be self-consistent (checked in the solver equivalence test) than consistent across versions.

user: ogrisel
issue_27484: state: open
title: Allow LogisticRegression with lbfgs solver to control `maxfun` parameter of solver
body: ### Describe the workflow you want to enable

Similarly to what is mentioned on https://github.com/scikit-learn/scikit-learn/issues/9273

> Training an MLP regressor (or classifier) using l-bfgs currently cannot run for more than (approx) 15000 iterations.
This artificial limit is caused by the call site to l-bfgs passing the MLP argument value "max_iters" to the argument for "maxfun" (maximum number of function calls), but not for "maxiter" (maximum number of iterations), so that no matter how large a number you pass as "max_iters" to train for MLP, the iterations are capped by the default value for maxiter (15000).

training a LogisticRegression regressor using l-bfgs currently cannot perform more than (approx) 15000 **function evaluations**.
This artificial limit is caused by the call site to l-bfgs passing the `LogisticRegression` argument value `max_iters` to the argument for `maxiter` (maximum number of iterations), but not for `maxfun` (maximum number of function evaluations), so that no matter how large a number you pass as "max_iters" to train for LogisticRegression, the function evaluations are capped by the default value for maxiter (15000, defined on https://github.com/scipy/scipy/blob/bf776169c753fff655200dc15ae26db95a083b02/scipy/optimize/_lbfgsb_py.py#L212).



### Describe your proposed solution

When calling the l-bfgs solver, set `maxfun` to be the same as `maxiter`, allow the user to control it.

As stated on [#9274](https://github.com/scikit-learn/scikit-learn/pull/9274) by @daniel-perry,
> Ideally you would want to pass in both a 'max_iter' and 'max_fun' argument to MLP, however 'max_fun' doesn't make sense for anything but l-bfgs, so using 'max_iter' to control both seems a reasonable compromise.

The same rational could be used here.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: vascosmota
labels: Needs Decision - API: Requires decision regarding API
issue_27483: state: open
title: Solve PCA via `np.linalg.eigh(X_centered.T @ X_centered)` instead of `np.linalg.svd(X_centered)` when `X.shape[1]` is small enough.
body: ### Describe the workflow you want to enable

Assuming that `X.shape[0] >> X.shape[1]` and `X.shape[1]` is small enough to materialize the covariance matrix `X.T @ X`, then using an eigensolver of the covariance matrix is much faster than the SVD of the centered data. See the proof of concept below:



### Describe your proposed solution

```python
>>> import numpy as np
>>> X = np.random.randn(int(1e6), 100)
>>> X -= X.mean(axis=0) # does not impact speed but required for PCA
>>> %time U, s, Vt = np.linalg.svd(X, full_matrices=False)
CPU times: user 22.2 s, sys: 1.59 s, total: 23.8 s
Wall time: 8.18 s
>>> %time eigenvals, eigenvecs = np.linalg.eigh(X.T @ X)
CPU times: user 81.1 ms, sys: 172 ms, total: 253 ms
Wall time: 255 ms
```

That's a 32x speed-up of the `"full"` solver. But it's also much faster than truncated randomized solver, even when `n_components` is quite low:

```python
>>> from sklearn.utils.extmath import randomized_svd
>>> %time U, s, Vt = randomized_svd(X, n_components=2)
CPU times: user 2.76 s, sys: 722 ms, total: 3.48 s
Wall time: 2.45 s
>>> %time U, s, Vt = randomized_svd(X, n_components=5)
CPU times: user 3.68 s, sys: 688 ms, total: 4.36 s
Wall time: 2.67 s
>>> %time U, s, Vt = randomized_svd(X, n_components=10)
CPU times: user 3.55 s, sys: 808 ms, total: 4.35 s
Wall time: 2.81 s
>>> %time U, s, Vt = randomized_svd(X, n_components=50)
CPU times: user 16.3 s, sys: 3.77 s, total: 20.1 s
Wall time: 11.9 s
```

And the results are the same (up to sign flips, hence the use of `np.abs`):

```python
>>> np.allclose(s ** 2, eigenvals[::-1])
True
>>> np.allclose(np.abs(eigenvecs[:, ::-1].T), np.abs(Vt))
True
```

Note that we might need to adapt the `svd_flip` helper (or even introduce `eigen_flip`) to do a proper deterministic sign swap for that strategy.

The SVD variant returns `U` which is useful for a cheap `fit_transform` but when `X.shape[0] >> X.shape[1]` and `X.shape[1]` is likely that `fit_transform()` implemented as `fit().transform()` will still be much faster with `eigh` on the covariance matrix.


Assuming we agree to add this strategy to scikit-learn, we have two options:
- introduce a new `svd_solver="covariance_eigh"` option, even if we don't actually perform an SVD anymore,
- alternatively we could make the `svd_solver="full"` option smart and use `eigh` on covariance whenever `X.shape[0] >> X.shape[1]`.

In either cases we might want to adjust the logic of `svd_solver="auto"` to use this whenever it makes sense (some benchmarking by adapting `benchmarks/bench_kernel_pca_solvers_time_vs_n_components.py` and `benchmarks/bench_kernel_pca_solvers_time_vs_n_samples.py` will be needed).

### Additional context

It might also be possible to do that for implicitly centered sparse data via a linear operator (as done in #18689) to form the covariance matrix (which is likely to be nearly dense when ``X.shape[0] >> X.shape[1]``). 

Also note that this should work with the Array API (see [array_api.linalg.eigh](https://data-apis.org/array-api/latest/extensions/generated/array_api.linalg.eigh.html#array_api.linalg.eigh)), and assuming `array-api-compat` provides a dask adaptor, this would be a very efficient way to do out-of-core (and even distributed) PCA on very large datasets (assuming `X.shape[1]` is small enough). It might be even more efficient than `IncrementalPCA`.
user: ogrisel
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_27462: state: open
title: Adding more functionality to DecisionBoundaryDisplay
body: This issue is to track the improvements that should be made to the `DecisionBoundaryDisplay`:

- [x] handle multiclass classification
    - [x] target a specific class: https://github.com/scikit-learn/scikit-learn/pull/27291
    - [ ] represent the max of probability
- [ ] allow to plot the decision boundary using a keyword without the need to call `DecisionBoundaryDisplay` twice
- [ ] Allow to pass `xlim/ylim` to overwrite the limit of the data seen in `from_estimator`. 
user: glemaitre
issue_27456: state: open
title: FEA Allow string input for pairwise distances
body: #### Reference Issues/PRs

closes #15932
closes #17991
closes #24674
supersedes #17991
supersedes #24674

#### What does this implement/fix? Explain your changes.

This allows the user to compute pairwise distance using a custom metric for non-metric data types (e.g. string or boolean).

#### Any other comments?

This PR just implements the changes from #24674 (which has stalled), including the last round of suggestions, and changes the name of the new parameter from `check_length_only` to `only_check_num_samples`.
user: julibeg
issue_27441: state: open
title: partial_dependence() computes conditional partial dependence
body: ### Describe the bug

For the case of correlated predictors (clearly highly common) the `sklearn.inspection.partial_dependence()` function gives different answers for `method` = "recursion" and `method` = "brute", see my [post](https://markusloecher.github.io/Partial-Dependence-Trees/) for elaborate examples.

I do not believe that this is intentional and should be fixed. Alternatively, it should be communicated clearly in the documentation that (i) the two methods are not equivalent for tree based algorithms, and (ii) that `method` = "recursion" actually computes the **conditional** $E[f(x_S,X_C)|X_S=x_s]$ instead of the (desired) **interventional** $E[f(x_S,X_C)| \mathbf{do}(X_S=x_s)]$ 

### Steps/Code to Reproduce

```python
from sklearn.tree import DecisionTreeRegressor
import numpy as np
import pandas as pd
from sklearn.inspection import PartialDependenceDisplay, partial_dependence

#Generate the data
#first X
N = 400; p0 = 0.5; p11 = 0.8; M=2
X = np.zeros((N,M)) # a matrix (N * M)
N1 = int(p0*N)
X[0:N1,0] = 1
X[0:int(p11*N1),1] = 1
X[N1:(N1+int((1-p11)*(N-N1))+1),1] = 1
df = pd.DataFrame(X, columns=["X0", "X1"])
#then Y
y = np.zeros(N)
y[(X[:,0] == 0) & (X[:,1] == 0)] = 0.3
y[(X[:,0] == 0) & (X[:,1] == 1)] = 0.7
y[(X[:,0] == 1) & (X[:,1] == 0)] = 0.9
y[(X[:,0] == 1) & (X[:,1] == 1)] = 0.1

df = pd.DataFrame(X, columns=["X0", "X1"])
df["y"] = y

model = DecisionTreeRegressor(max_depth=2, max_features = 2)
model.fit(df[["X0","X1"]], y);

features = ["X1"]
for f in features:
    pdp_interventional = partial_dependence(model, df[["X0","X1"]], f,method = "brute")
    pdp_conditional = partial_dependence(model, df[["X0","X1"]], f,method = "recursion")
    
    print(f, "brute (interventional):", pdp_interventional['average'])
    print(f, "recursion (conditional):", pdp_conditional['average'])

pdp_interventional['average'] == pdp_conditional['average']
```

### Expected Results

We would like the two methods to yield the same pdp values, so the last line should yield

`array([[True, True]])`

### Actual Results

Instead they are different. The print statements yield
_X1 brute (interventional): [[0.6 0.4]]
X1 recursion (conditional): [[0.42 0.22]]_
and the last line yields
`array([[False, False]])`

### Versions

```shell
System:
    python: 3.9.0 (v3.9.0:9cf6752276, Oct  5 2020, 11:29:23)  [Clang 6.0 (clang-600.0.57)]
executable: /Users/loecherm/.pyenv/versions/aug_hs_env/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.2.0
          pip: 23.0.1
   setuptools: 49.2.1
        numpy: 1.23.5
        scipy: 1.10.0
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.5.3
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

user: markusloecher
issue_27439: state: open
title: _safe_indexing being the only *private* thing listed in classes.rst
body: `_safe_indexing` is the only _private_ tool we have listed in `classes.rst`.

`_safe_indexing` was also mentioned in https://github.com/scikit-learn/scikit-learn/issues/15801, but this issue is to have the discussion about this very limited scope.

We should really either remove it from the docs, or make it public.

cc @glemaitre @thomasjpfan 
user: adrinjalali
issue_27422: state: open
title: HalvingGridSearchCV giving non-optimal results with min_resources='exhaust'
body: ### Describe the bug

I am using `HalvingGridSearchCV` with 160 combinations and 1050 samples. When I use `min_resources` with `'exhaust'`, I get 6 iterations with the last iteration including 5 candidates and 640 samples. The process starts with 20 samples.

When I set `min_resources=30`, I also get 6 iterations with the last iteration having 5 candidates, but with 960 samples.

### Steps/Code to Reproduce

```python
halving_cv = HalvingGridSearchCV(
    knn_model, knn_grid,  #our model and the parameter combos we want to try
    scoring="roc_auc",  #could alternatively choose f1, accuracy or others
    n_jobs=-1,  #use all available cpus
    min_resources=30,  #"exhaust" sets this to 20, which is non-optimal. Possible bug in algorithm.
    factor=2,  #double samples and take top half of combos on each iteration
    cv=5, random_state=1234,
    refit=True,  #remembers the best combo and gives us back that model already trained and ready for testing
)

grid_result = halving_cv.fit(x_train, y_train)
```

### Expected Results

My understanding is that by using 'exhaust', the algorithm should figure out that 30 is a better value for min_resources. Instead it sets it to 20.

My ideal case would be for the last iteration to use all 1050 samples with the 5 candidates but I see no way to set parameters to make this happen.

My question is whether there is a bug in the min_resources=exhaust algorithm.

### Actual Results

With min_resources=30:
```
310	5	960	{'algorithm': 'auto', 'n_neighbors': 15, 'p': 1, 'weights': 'uniform'}	0.804883
311	5	960	{'algorithm': 'brute', 'n_neighbors': 5, 'p': 1, 'weights': 'uniform'}	0.793407
312	5	960	{'algorithm': 'ball_tree', 'n_neighbors': 5, 'p': 1, 'weights': 'uniform'}	0.789650
313	5	960	{'algorithm': 'auto', 'n_neighbors': 5, 'p': 1, 'weights': 'uniform'}	0.789410
314	5	960	{'algorithm': 'kd_tree', 'n_neighbors': 5, 'p': 1, 'weights': 'uniform'}	0.789410
```

### Versions

```shell
System:
    python: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
executable: /usr/bin/python3
   machine: Linux-5.15.109+-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.2
   setuptools: 67.7.2
        numpy: 1.23.5
        scipy: 1.11.2
       Cython: 3.0.2
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/numpy.libs/libopenblas64_p-r0-742d56dc.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: Zen

       user_api: openmp
   internal_api: openmp
    num_threads: 2
         prefix: libgomp
       filepath: /usr/local/lib/python3.10/dist-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Zen

       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/cvxopt.libs/libopenblasp-r0-5c2b7639.3.23.so
        version: 0.3.23
threading_layer: pthreads
   architecture: Zen
```
```

user: fickas
labels: Needs Reproducible Code: Issue requires reproducible code
issue_27412: state: open
title: FIX Make ROC AUC score consistent
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes https://github.com/scikit-learn/scikit-learn/issues/24381

See also #27047 

#### What does this implement/fix? Explain your changes.
 - replaced ValueError with UndefinedMetricWarning
 - replaced -0.0 with 0.0
 - updated tests

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: glevv
issue_27388: state: open
title: [MRG] ENH Add support for viewing marginal distributions in partial dependence plots
body: #### Reference Issues/PRs
Closes #27528. Also discussed in #19410. 

#### What does this implement/fix? Explain your changes.
This PR adds support for plotting feature distributions with partial dependence plots. It allows users to plot histograms, boxplots and scatterplots for both one and two-way partial dependence plots. 

#### Any other comments?
Some observations and topics for discussion:
1. For one-way scatter plots, I added support for plotting actual target values and the predicted target values. Actual values from binary classification don't look pleasing. 
2. I clip the histogram and boxplots to the min and max of the pd values. I was torn on this decision because one of the advantages of viewing the feature distributions is seeing the entire distribution.
3. Do we need both histogram and boxplot?

Here are some examples from the auto_example tutorial:

```python
from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestRegressor
from sklearn.inspection import PartialDependenceDisplay
from sklearn.model_selection import train_test_split

import matplotlib.pyplot as plt
import pandas as pd


X, y = load_diabetes(as_frame=True, return_X_y=True)
X["sex"] = pd.get_dummies(X["sex"], drop_first=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0, train_size=0.6)

rf = RandomForestRegressor(
    max_depth=5, random_state=0
)
rf.fit(X_train, y_train)

_, ax = plt.subplots(figsize=(6, 4), constrained_layout=True)
display = PartialDependenceDisplay.from_estimator(
    rf, X_test, ["bp", ("bmi", "bp")], extra_plots=["boxplot", "hist"], ax=ax
)
```
![image](https://github.com/scikit-learn/scikit-learn/assets/9151717/1e1b66d7-7b12-4fed-bdd6-61579d910329)

```python
_, ax = plt.subplots(figsize=(6, 4), constrained_layout=True)
display = PartialDependenceDisplay.from_estimator(
    rf,
    X_test,
    ["s5", ("bmi", "bp")],
    extra_plots="scatter",
    extra_plots_kw={"scatter": {"alpha": 0.25}},
    y=y_test,
    ax=ax,
)
```
![image](https://github.com/scikit-learn/scikit-learn/assets/9151717/1e042c77-8f2a-47b5-8f7b-589e27958ffb)


user: arvkevi
issue_27381: state: open
title: ENH Array API support for LabelEncoder
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Towards #26024
Related to #27369

#### What does this implement/fix? Explain your changes.
- Adds Array API support for LabelEncoder including all the inner functions that it uses.

#### Any other comments?
CC: @betatim @ogrisel 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: OmarManzoor
issue_27378: state: open
title: Replace _inplace_contiguous_isotonic_regression by scipy.optimize.isotonic_regression
body: With the upcomping scipy 1.12, there will be the new function `scipy.optimize.isotonic_regression`, see https://github.com/scipy/scipy/pull/17722.

We can start using this function (which is a bit faster than our own method) and replace ours completely once scipy 1.12 is the minimum scipy version.
user: lorentzenchr
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_27377: state: open
title: TST Uncomment test in `test_sag.py`
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.


#### Any other comments?
fyi @glemaitre 
It was using tol, which is unsupported for sag, so I changed it.
https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/linear_model/_ridge.py#L857
Also max_iter should be equal None, which will become 1000 later, so I created new variable for it.

There were some floating precision problems so I changed accuracy of asserts.
example:
```
Arrays are not almost equal to 3 decimals

Mismatched elements: 3 / 10 (30%)
Max absolute difference: 0.00369502
Max relative difference: 0.00583289
x: array([-0.466, -0.971,  0.653,  0.319, -1.614,  0.306,  0.711,  0.068,
      -0.238, -0.637])
y: array([-0.468, -0.969,  0.653,  0.319, -1.616,  0.305,  0.711,  0.068,
      -0.239, -0.633])
```

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Tialo
issue_27375: state: open
title: Adding documentation about related library: Concrete ML, for privacy preserving ML
body: ### Describe the issue linked to the documentation

We would like to add Concrete ML in the documentation. Concrete ML is an open-source package, providing privacy-preserving ML, thanks to so-called fully homomorphic encryption. Concrete ML contains so-called built-in models which are very very close to scikit-learn, XGB and skorch.  Eg, see https://docs.zama.ai/concrete-ml/built-in-models/linear 

We've started to discuss with some of you, who advised me to go for an issue. 

I would propose: 
- [x] adding a category Privacy Preserving Machine Learning in ./doc/related_projects.rst, and adding Concrete ML inside --- done, this was https://github.com/scikit-learn/scikit-learn/pull/27376
- [x] adding a category Privacy Preserving Machine Learning in examples, and copy some of the examples of https://github.com/zama-ai/concrete-ml/tree/main/docs/advanced_examples, or make new ones (what you prefer!) [not to be done]
- [ ] maybe propose a blog post, if you guest us. Will be done soon, https://github.com/zama-ai/concrete-ml-internal/issues/4033

and we are actually open to lot of suggestions.

What do you think?

### Suggest a potential alternative/fix

Adding some content as proposed in the description.
user: bcm-at-zama
issue_27373: state: open
title: QuantileTransformer's default subsampling introduces artefacts for unbounded distributions
body: ### Describe the bug

The default behaviour of subsampling in the QuantileTransformer introduces artefacts when the input data originates from an unbounded distribution and the transformed dataset is (significantly) larger than the fitted one.

For any unbounded distribution the largest (smallest) percentiles will in expectation be underestimated (overestimated).
This effect is non-trivial for the selected subsample size of 10_000.

Here we draw a Multivariate Normal distribution of size 100_000
![mvn_input](https://github.com/scikit-learn/scikit-learn/assets/18510572/65676640-0674-46c8-a712-6e73e3a45b93)

Running `QuantileTransformer(output_distribution="normal").fit_transform(X)` i.e. with subsampling results in the below plot. Note the artefacts in the corners and the noisy marginal distributions.
![mvn_qt_defaults](https://github.com/scikit-learn/scikit-learn/assets/18510572/ddb42fb5-252f-4d62-80f4-2bbc2d12abf8)

Turning off the subsampling by specifying the subsampling size to be >= than the size of `X`. (this is undocumented btw)
`QuantileTransformer(output_distribution="normal", subsample=X.shape[0] + 1).fit_transform(X)`
![mvn_qt_no_sampling](https://github.com/scikit-learn/scikit-learn/assets/18510572/461601d5-9562-4230-a6f3-9a1081460cd2)

The effect of subsampling can be easily demonstrated by computing the difference over the subsample percentiles and the percentiles as you would estimate over the complete sample.
```Python
percentiles = np.arange(1, 100)
x = rng.normal(100_000)
xs = rng.choice(x, size=10_000, replace=False)
deltas = np.nanpercentile(x, percentiles) - np.nanpercentile(xs, percentiles)
```
The plot below shows the mean delta per percentile over 100_000 simulations of the above:
![standard_normal_subsample_delta](https://github.com/scikit-learn/scikit-learn/assets/18510572/c333c349-25f3-4386-8d2d-15149528d8a5)


### Steps/Code to Reproduce

```python
import numpy as np
import pandas as pd
import scipy.stats as sps
import seaborn as sns
from sklearn.preprocessing import QuantileTransformer

import matplotlib
import matplotlib.pyplot as plt

matplotlib.rcParams['pdf.fonttype'] = 42
matplotlib.rcParams['ps.fonttype'] = 42
plt.rcParams['text.color'] = 'black'
plt.rcParams['figure.max_open_warning'] = 0
colors = [i['color'] for i in plt.rcParams['axes.prop_cycle']]
markers = ['o', 's', 'p', 'x', '^', '+', '*', '<', 'D', 'h', '>']


def plot_joint(X, ax=None):
    df = pd.DataFrame(X, columns=["x", "y"])
    axs = sns.JointGrid(data=df, x='x', y='y')
    axs.ax_joint.scatter(data=df, x='x', y='y', c=colors[0], marker='x', s=0.3)
    cnt, bins, _ = axs.ax_marg_x.hist(df['x'], bins=100, color=colors[0])
    cnt, bins, _ = axs.ax_marg_y.hist(df['y'], bins=100, color=colors[0], orientation='horizontal')
    axs.ax_joint.set_xlabel('')
    axs.ax_joint.set_ylabel('')
    return axs


def mvn_sample(rng, size=100_000, mu_x = 0., mu_y = 0., sigma_x = 1., sigma_y = 1., rho = 0.7):
    return rng.multivariate_normal(
        [mu_x, mu_y],
        [
            [sigma_x * sigma_x, rho * sigma_x * sigma_y],
            [rho * sigma_x * sigma_y, sigma_y * sigma_y]
        ],
        size=size
    )

rng = np.random.Generator(np.random.PCG64DXSM(753242))
X = mvn_sample(rng)

plot_joint(X).fig.show()
plot_joint(QuantileTransformer(output_distribution="normal").fit_transform(X)).fig.show()
plot_joint(QuantileTransformer(output_distribution="normal", subsample=X.shape[0] + 1).fit_transform(X)).fig.show()
```

### Expected Results

No artefacts introduced by subsampling, subsampling should not be on by default.

### Actual Results

See above.

### Versions

```shell
System:
    python: 3.11.3 (main, May  1 2023, 15:33:25) [Clang 14.0.3 (clang-1403.0.22.14.1)]
executable: ~/.pyenv/versions/3.11.3/envs/SKB/bin/python3.11
   machine: macOS-13.5.2-arm64-arm-64bit

Python dependencies:
      sklearn: 1.3.0
          pip: 22.3.1
   setuptools: 65.5.0
        numpy: 1.25.2
        scipy: 1.11.1
       Cython: None
       pandas: 2.0.3
   matplotlib: 3.7.2
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: libomp
       filepath: ~/.pyenv/versions/3.11.3/envs/SKB/lib/python3.11/site-packages/sklearn/.dylibs/libomp.dylib
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libopenblas
       filepath: ~/.pyenv/versions/3.11.3/envs/SKB/lib/python3.11/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: armv8

       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libopenblas
       filepath: ~/.pyenv/versions/3.11.3/envs/SKB/lib/python3.11/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: armv8
```

user: RUrlus
issue_27369: state: open
title: ENH Array API support for f1_score
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Towards #26024 

#### What does this implement/fix? Explain your changes.
- Adds array api support for f1_score and the functions related to it.

#### Any other comments?
CC: @ogrisel @betatim 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: OmarManzoor
issue_27368: state: open
title: allow uniform intialization for BayesianGaussianMixture with dirichlet_process wight prior
body: ### Describe the workflow you want to enable

I would like to have a deterministic version of the Bayesian Gaussian Mixture Model

### Describe your proposed solution

I propose allowing for unifrom responsibility initialization. This is possible when using the dirichlet process weight prior, because it is inherently asymmetric with the only assuption that there is an order to the cluster sizes

### Describe alternatives you've considered, if relevant

random initialization is non-deterministic which tends to introduce unwanted noise in the process

### Additional context

I have manually hacked in uniform initialzation, which is rather simple. it should likely be forbidden if the prior is not a dirichlet process but a symmetric dirichlet distribution
user: maddanio
issue_27357: state: open
title: DOC improve metadata routing example
body: #### What does this implement/fix? Explain your changes.
This PR aims to improve the Metadata Routing example for clarity and readability.  I've also enhanced two error messages in `sklearn/utils/_metadata_requests.py` to make them better understandable for inexperienced users.

I feel that this example is already pretty comprehensible and contains all the information needed (though it's a slow read). I've mainly worked on wording and variable naming.

I'm looking forward to your reviews, @adrinjalali and @glemaitre. :)

#### Any other comments?
Edit: It was a draft, but now it's ready to be a real PR.
user: StefanieSenger
issue_27349: state: open
title: SimpleImputer fails for columns with pandas string type
body: ### Describe the bug

The SimpleImputer class with strategy="most_frequent" fails during fit when one of the columns of the input dataframe is of type string.

The error happens here https://github.com/scikit-learn/scikit-learn/blob/d8e131c1640f13954b2dd5f0cc3b50b82a05a554/sklearn/utils/fixes.py#L58-L59

because (use X from the example below) `np.array(X.iloc[:, 0]) == np.array(X.iloc[:, 0])` returns a bool (`False`) instead of an array of bools.

### Steps/Code to Reproduce

```python
from sklearn.impute import SimpleImputer
import numpy as np
import pandas as pd

X = pd.DataFrame([['A'], ['B'], [np.nan]], dtype="string")
print(X.dtypes)

SimpleImputer(strategy="most_frequent").fit(X)
```

### Expected Results

No errors thrown.



### Actual Results

```
0    string[python]
dtype: object

Traceback (most recent call last):
  File "C:\Users\Mavs\Documents\Python\ATOM\venv310\lib\site-packages\IPython\core\interactiveshell.py", line 3460, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-7b70f7e70352>", line 8, in <module>
    SimpleImputer(strategy="most_frequent").fit(X)
  File "C:\Users\Mavs\Documents\Python\ATOM\venv310\lib\site-packages\sklearn\base.py", line 1151, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "C:\Users\Mavs\Documents\Python\ATOM\venv310\lib\site-packages\sklearn\impute\_base.py", line 405, in fit
    self.statistics_ = self._dense_fit(
  File "C:\Users\Mavs\Documents\Python\ATOM\venv310\lib\site-packages\sklearn\impute\_base.py", line 488, in _dense_fit
    mask = missing_mask.transpose()
AttributeError: 'bool' object has no attribute 'transpose'
```

### Versions

```shell
System:
    python: 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)]
executable: C:\Users\Mavs\Documents\Python\ATOM\venv310\Scripts\python.exe
   machine: Windows-10-10.0.19045-SP0
Python dependencies:
      sklearn: 1.3.0
          pip: 23.2.1
   setuptools: 59.8.0
        numpy: 1.23.5
        scipy: 1.10.1
       Cython: 0.29.30
       pandas: 2.0.3
   matplotlib: 3.7.2
       joblib: 1.3.1
threadpoolctl: 3.1.0
Built with OpenMP: True
threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: C:\Users\Mavs\Documents\Python\ATOM\venv310\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 16
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\Mavs\Documents\Python\ATOM\venv310\Lib\site-packages\numpy\.libs\libopenblas.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.gfortran-win_amd64.dll
        version: 0.3.20
threading_layer: pthreads
   architecture: Zen
    num_threads: 16
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\Mavs\Documents\Python\ATOM\venv310\Lib\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: Zen
    num_threads: 16
```

user: tvdboom
issue_27344: state: open
title: ENH Improve the efficiency of QuantileTransformer
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
See #27263

#### What does this implement/fix? Explain your changes.
The original subsampling in QuantileTransformer was done column by column, I think subsampling can just do once to improve efficiency. Also, I remove the for-loop and transpose while computing the quantiles.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: xuefeng-xu
issue_27342: state: open
title: ENH Add `pos_label` parameter to `TargetEncoder`
body: ### Describe the workflow you want to enable

Add a `pos_label` parameter to `TargetEncoder` to enable the user to specify which label should be the positive class when the target is binary.

### Describe your proposed solution

Add a `pos_label` parameter that is passed to the `LabelBinarizer` `pos_label` parameter.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Ref: https://github.com/scikit-learn/scikit-learn/pull/26674#discussion_r1317446826

cc @ogrisel 
user: lucyleeow
issue_27340: state: open
title: A welcome bot
body: Shall we add a welcome bot with a nice and pleasant message as:
![image](https://github.com/scikit-learn/scikit-learn/assets/208217/b0ba57aa-8010-420d-9712-3c832f076b9f)
https://github.com/scikit-learn/blog/pull/170

It gave me a cheesy feel-good moment, and I like these.

More seriously, I think that such a both can help
- Keep positive vibes
- Manage expectations / give early hints about how to move a PR forward
user: GaelVaroquaux
issue_27339: state: open
title: RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar?
body: While working on the representation imposed by NEP51, I found out that we recently made the `accuracy_score` to return a Python scalar while, up-to-now, other metric are returning NumPy scalar.

This change was made due to the array API work:

https://github.com/scikit-learn/scikit-learn/blob/b0da1b7706054f0b78f0a0582a9362a188e1fa38/sklearn/utils/_array_api.py#L448-L454

I assume that we are getting to an intersection where we should make the output of our metrics consistent but also foresee potential requirements: as the comment indicate, calling `float()` will be a sync point but it might not be the best strategy for lazy computation.

This RFC is a placeholder to discuss what strategy we should be implementing.
user: glemaitre
issue_27338: state: open
title: Feature Request: Out-of-Vocabulary (OOV) Token Handling in CountVectorizer
body: ### Describe the workflow you want to enable

Users of CountVectorizer frequently encounter situations where, during the transformation of test data or new unseen data, there are words that the vectorizer hasn't seen during the training (fit) phase. In such situations, the default behavior is to ignore these words. However, for many NLP applications (like sentiment analysis, named entity recognition, etc.), having a representation for these OOV words is crucial. Hence, an inbuilt mechanism to handle OOV words by representing them with a special token would be beneficial.

### Describe your proposed solution

1. Additional Parameter: Introduce a new optional parameter in CountVectorizer called oov_token which defaults to None.

When oov_token is None, the vectorizer behaves as it currently does.
When oov_token is provided (e.g., <OOV>), any word encountered during the transform phase that is not in the learned vocabulary gets represented by this token.
2. Vocabulary Inclusion: The oov_token, if provided, should be part of the vocabulary, and its position/index in the vocabulary should be consistent.

3. OOV Token Counting: The count of the OOV token should reflect the total count of all OOV words in a document.

### Describe alternatives you've considered, if relevant

Custom Vectorizer: Users currently have to extend the CountVectorizer class to handle OOV, which adds extra overhead. An inbuilt feature would be more efficient and user-friendly.


### Additional context

_No response_
user: vimarshc
issue_27337: state: open
title: Add kNN outlier and Local Outlier Probabilities (LoOP)
body: #### Reference Issues/PRs

Alternative implementation to #10121, with a unit test against the original implementation, as I am not sure if @vc1492a is still working on his pull request. This also adds kNN outlier using a consistent API, previously proposed in #24697 #24574, although it has been discussed that the user could easily do this themselves (the effective code is, unsurprisingly, a one-liner, everything else is boilerplate). Nevertheless, there is *value* in having several methods easy to use in a uniform API for easier testing and benchmarking.

#### What does this implement/fix? Explain your changes.

This branch implements LoOP (Local Outlier Probabilities), an unsupervised outlier detection approach that's similar to LOF (Local Outlier Factor) but that normalizes scores in the range [0, 1].

> Kriegel, H. P., Kröger, P., Schubert, E., & Zimek, A. (2009, November). LoOP: local outlier probabilities. In Proceedings of the 18th ACM conference on Information and knowledge management (pp. 1649-1652).

This method currently has [638 citations on Google scholar](https://scholar.google.com/scholar?q=LoOP%3A%20local%20outlier%20probabilities), so its probably worth inclusion.

PyOD has been hoping/waiting for this to be included with sklearn since 2018: https://github.com/yzhao062/pyod/issues/8

I also added two variants of classic nearest-neighbor outlier detection (>3000 citations):

> Ramaswamy, S., Rastogi, R., & Shim, K. (2000, May). Efficient algorithms for mining outliers from large data sets. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data (pp. 427-438).

and (>1000 citations)

> Angiulli, F., & Pizzuti, C. (2002, August). Fast outlier detection in high dimensional spaces. In European conference on principles of data mining and knowledge discovery (pp. 15-27).

All three qualify by https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
as  "well-established algorithm"s of "wide use and usefulness".

#### Any other comments?

The code is a straight-forward copy of the LOF implementation, I did not attempt to make a uniform wrapper around this, but rather get the key functionality in and tested first. I believe abstraction is better done when some more outlier detection methods have been added (see "Local outlier detection reconsidered: a generalized view on locality with applications to spatial, video, and network outlier detection." for some ideas on how to abstract this class of methods).

I am not convinced of the current design in sklearn to use negative scores, such as the negative_local_outlier_factor in LOF, but this also goes beyond the scope of this pull request. I understand the original idea of having OCSVM and LOF use the same ranking, and OCSVM uses -1 for out-of-distribution and +1 for in-distribution samples because of the way the optimization problem is formulated. But for LoOP this breaks the notion of having an "outlier probability". So this may well be an example of trying to abstract between OCSVM and LOF too early... Nevertheless, I tried to stick as close as possible to the current decisions of sklearn, these should be resolved in a separate issue.

For LoOP, it would also be possible to use "inlier probabililties", i.e., "1-loop" instead of "0-loop", or even "1-2*loop" to get closer to the -1..+1 semantics of OCSVM, but I do not think this is actually good.

Maybe the `@property` mechanism could be used to provide some backwards compatibility with the current "negative score" design, and eventually move to using scores closer to the literature, i.e., high values indicating outlierness. If I could rewrite all the literature, I would prefer scores to measure normality, but that is neither what is common in literature, nor what users expect.

I would also suggest to move them to a separate package. It is odd to have outlier detectors distributed to the sklearn.svm and sklearn.neighbors packages, while clustering algorithms live in sklearn.cluster - there should eventually be a `sklearn.outlier` package?
user: kno10
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_27332: state: open
title: Raise ValueError instead of RuntimeWarning for LinearModelLoss
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Fixes #27016  


#### What does this implement/fix? Explain your changes.

Fixes an issue where ```PoissonRegressor``` (and other regressors) encounter a ```RuntimeWarning```, without any suggestion of the cause or ways to fix.

The fix captures the ```RuntimeWarning``` and raises a ```ValueError``` instead, in the case where the gradient vector contains invalid entries (e.g. inf, null), during a matmul. It also has a message suggesting scaling of features or target variable, or to use another solver.

#### Any other comments?

So far I have just tested that this exception/message occur on the same test data where I encountered the issue (see the issue #27016).

I have included the same change for the case where the loss is multiclass, but haven't directly tested this on a multiclass problem.

The change caused some pre-existing pytests to start failing, due to warning that would have been raised being replaced by a ValueError as part of the issue. I have included a change to scale the X dataset so that this error doesn't occur to get those tests passing.

Questions:

1. Is the update to the pytest valid? And if it is, should it be part of this PR or a separate one?
2. Could throwing a ValueError rather than just a warning break some pipelines/uses of linear loss that are currently 'working'? ('Working' in the sense that there is only a RuntimeWarning, not an error/exception) If so, is it possible (or better even) to instead just update the message on the RuntimeWarning?
3. There are still 3 checks that are failing - how could they be resolved?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: akaashp2000
issue_27321: state: open
title: Hwy dispatch alt
body: #### Reference Issues/PRs

#### What does this implement/fix? Explain your changes.
Dynamic dispatch via `HWY_DYNAMIC_POINTER`


#### Any other comments?

user: Micky774
issue_27313: state: open
title: Implementation of Order Recursive Matching Pursuit
body: ### Describe the workflow you want to enable

The sparse linear regression problem is the NP-hard problem of performing ordinary L² linear regression with the catch that at most a fixed number of the resulting coefficients can be non-zero (or the variation in which one wants as few non-zero columns as possible while reaching a given score). Scikit-learn includes a greedy approximation algorithm for solving this problem in the form of [`sklearn.linear_model.OrthogonalMatchingPursuit`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.OrthogonalMatchingPursuit.html#sklearn.linear_model.OrthogonalMatchingPursuit) (OMP).

Many variations on this algorithm exist; one of them (which dates back [at least to the 70s](https://www.jstor.org/stable/1267353)) is "Order Recursive Matching Pursuit" (ORMP). Like OMP, ORMP is an iterative algorithm which picks one column from the input at a time, while maintaining the current residual. In OMP, one picks the column whose angle with the current residual is minimal, whereas in ORMP, one chooses the column resulting in the minimal residual. That is, while the approach is still greedy, it's somewhat more to the point, choosing the column leading to the minimal objective in each step. A priori, the cost of this approach is increased computational time, but with proper care, it is possible to make a [simple implementation](https://github.com/Kvantify/ormp/blob/master/src/ormp/impl_np.py) that rivals the runtime performance of the existing OMP implementation, and is faster on the instances that motivated the work.

### Describe your proposed solution

In our company, we needed fit accuracy greater than what could be achieved with OMP and thus implemented ORMP in a handful of different ways. The implementations are available on https://github.com/Kvantify/ormp. As I believe it fits the inclusion criteria, these implementations could be lifted into scikit-learn more or less directly, some parts of it more relevant than others: I imagine we don't want a JAX dependency in scikit-learn, for example, but note that even the pure NumPy implementation appears to be generally faster than scikit-learn's existing OMP, while delivering the promised score improvements on typical (but far from all) problem instances. See the README in that repository for [a few crude performance benchmarks](https://github.com/Kvantify/ormp#choosing-an-implementation) and [notes on a result quality comparison with OMP](https://github.com/Kvantify/ormp#comparison-with-orthogonal-matching-pursuit).

We can submit a pull request if you are interested.
user: fuglede
issue_27305: state: open
title: Monotonicity constraints for GradientBoostingClassifier and GradientBoostingRegressor
body: ### Describe the workflow you want to enable

As a follow-up of #13649, I'd like to use
```python
GradientBoostingClassifier(monotonic_cst=...)
```
same as in `HistGradientBoostingClassifier` and int `RandomForestClassifier`.

### Describe your proposed solution

Add `monotonic_cst` to `GradientBoostingClassifier` and `GradientBoostingRegressor`.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: lorentzenchr
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_27259: state: open
title: New clustering metrics
body: ### Describe the workflow you want to enable

Scikit-learn defines three popular metrics for evaluating clustering performance when there are no ground-truth cluster labels: [sklearn.metrics.silhouette_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html), [sklearn.metrics.calinski_harabasz_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.calinski_harabasz_score.html) and [sklearn.metrics.davies_bouldin_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.davies_bouldin_score.html). But there are lots of others, and it's previously been [discussed](https://github.com/scikit-learn/scikit-learn/discussions/21164) whether to integrate more into scikit-learn.

### Describe your proposed solution

I've implemented four relatively popular ones, using the same interface and code style as in https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/cluster/_unsupervised.py. Would there be interest in integrating these into scikit-learn?

```python
import numpy as np
from itertools import combinations
from sklearn.utils import check_X_y
from sklearn.metrics.cluster._unsupervised import check_number_of_labels
from sklearn.preprocessing import LabelEncoder


def log_ss_ratio(X, labels):
    X, labels = check_X_y(X, labels)
    le = LabelEncoder()
    labels = le.fit_transform(labels)

    n_samples, _ = X.shape
    n_labels = len(le.classes_)

    check_number_of_labels(n_labels, n_samples)

    extra_disp, intra_disp = 0.0, 0.0
    mean = X.mean(axis=0)
    for k in range(n_labels):
        cluster_k = X[labels == k]
        mean_k = cluster_k.mean(axis=0)
        extra_disp += len(cluster_k) * ((mean_k - mean) ** 2).sum()
        intra_disp += ((cluster_k - mean_k) ** 2).sum()

    return np.log(extra_disp / intra_disp)


def ball_hall(X, labels):
    X, labels = check_X_y(X, labels)
    le = LabelEncoder()
    labels = le.fit_transform(labels)

    n_samples, _ = X.shape
    n_labels = len(le.classes_)

    check_number_of_labels(n_labels, n_samples)

    sum_mean_dispersions = 0

    for k in range(n_labels):
        cluster_k = X[labels == k]
        mean_k = cluster_k.mean(axis=0)
        intra_disp_cluster = ((cluster_k - mean_k) ** 2).sum()
        sum_mean_dispersions += intra_disp_cluster / len(cluster_k)

    return sum_mean_dispersions / n_labels


def banfeld_raftery(X, labels):
    X, labels = check_X_y(X, labels)
    le = LabelEncoder()
    labels = le.fit_transform(labels)

    n_samples, _ = X.shape
    n_labels = len(le.classes_)

    check_number_of_labels(n_labels, n_samples)

    br_index = 0
    for k in range(n_labels):
        cluster_k = X[labels == k]
        mean_k = cluster_k.mean(axis=0)
        intra_disp_cluster = ((cluster_k - mean_k) ** 2).sum()
        br_index += len(cluster_k) * \
                    np.log(intra_disp_cluster / len(cluster_k))

    return br_index


def ray_turi(X, labels):
    X, labels = check_X_y(X, labels)
    le = LabelEncoder()
    labels = le.fit_transform(labels)

    n_samples, _ = X.shape
    n_labels = len(le.classes_)

    check_number_of_labels(n_labels, n_samples)

    intra_disp = 0.0

    cluster_means = []
    for k in range(n_labels):
        cluster_k = X[labels == k]
        mean_k = cluster_k.mean(axis=0)
        intra_disp += ((cluster_k - mean_k) ** 2).sum()
        cluster_means.append(mean_k)

    min_cluster_mean_diff = min(
        ((mean_i - mean_j) ** 2).sum()
        for mean_i, mean_j in combinations(cluster_means, 2))

    return intra_disp / (min_cluster_mean_diff * n_samples)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: Wainberg
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_27232: state: open
title: ENH Remove hardcoded device choice in _weighted_sum
body: #### What does this implement/fix? Explain your changes.
Some Array API compatible libraries do not have a device called 'cpu'. Instead to deal with the fact that some devices (e.g. MPS) do not have `float64` support, we try and detect the lib+device combination that does not support `float64`.

I also made it so that if weights are passed that array is passed to `get_namespace` so that both arrays have to be in the same namespace.

In general there is no generic way of specifying a device that all libraries understand. This means the string `'cpu'` might or might not be understood. In addition there are libraries like cupy that do not even have a CPU device (no matter what you call it).

I was also wondering what would happen if we have the scores on the CPU, but the weights on a different device.

#### Any other comments?

This is related to/inspired by reading/xref #27137. That PR also adds tests for `_weighted_sum` which is why I've not added any here.

For my education: why do we convert the type to float if it isn't? For example if we are doing a sum of an array of integers, without weights, then it seems more natural to stay with `int`s (to avoid the problem that some `int`s can't be represented by floats, etc).

I think this shows that we need to figure out a solution to the lack of CI :-/
user: betatim
issue_27200: state: open
title: Implementation of Robust Random Cut Forest (RRCF) Algorithm
body: ### Describe the workflow you want to enable

Enable users to perform robust anomaly detection using the Robust Random Cut Forest (RRCF) algorithm within the scikit-learn library.

### Describe your proposed solution

## Proposed Solution

I suggest the integration of the Robust Random Cut Forest (RRCF) [[paper](http://proceedings.mlr.press/v48/guha16.pdf)] algorithm into the scikit-learn library. The RRCF algorithm is a natural extension of the Isolation Forest. It presents improved capabilities in handling noisy data, outliers, and stream data scenarios.

The core idea behind RRCF is akin to the Isolation Forest—building binary search trees using random cuts in the input data space. The key distinction lies in the _probability of selecting a cut dimension being proportionate to the dimension's length_. This unique characteristic renders RRCF particularly suitable for scenarios where the Isolation Forest may not perform optimally.

## Implementation

The underlying tree and forest structure of RRCF closely resembles that of the existing iForest, as previously mentioned. The primary difference arises in the computation of anomaly scores, specifically within the `decision_function`.

Additionally, to cater to stream data scenarios, the implementation could (and it should) incorporate methods for efficient insertion and deletion of individual points, as mentioned in the paper.

### Existing Implementations

A pre-existing Python implementation of RRCF can be found at https://github.com/kLabUM/rrcf. This implementation appears to be well-structured and accurate, making it a potential reference for scikit-learn's integration.

I would greatly appreciate your consideration of this proposal. Thank you.

On a side note, please don't hesitate to point out any mistakes/enhancements in the proposal. I'll be glad to assist.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This past summer, I had the opportunity to work with anomaly detection algorithms, and among them, I had the chance to closely explore the Robust Random Cut Forest (RRCF). Interestingly, I discovered that while scikit-learn already has Isolation Forest algorithm, RRCF implementation is still missing. This revelation has sparked a motivating drive within me to advocate for the integration of RRCF.

Although it is plausible that this discussion on this topic might have already been occurred, but I was not able to find anything related to RRCF in the issues section or in [scikit-learn-contrib](https://github.com/scikit-learn-contrib/scikit-learn-contrib) except in [this comment](https://github.com/scikit-learn/scikit-learn/issues/17307#issuecomment-632795440) (where the context is kind of different).


user: gjain-7
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_27186: state: open
title: BUG (maybe) wrong node bound spread in KernelDensity
body: ### Describe the bug

https://github.com/scikit-learn/scikit-learn/blob/a5620f45614ac3f849c430f53146a66319e4908b/sklearn/neighbors/_binary_tree.pxi.tp#L2114-L2116

https://github.com/scikit-learn/scikit-learn/blob/a5620f45614ac3f849c430f53146a66319e4908b/sklearn/neighbors/_binary_tree.pxi.tp#L2121-L2123

These lines do not seem to be correct. The right-hand side seems to be node log max bound but not node log bound spread. Should they instead be

```python
node_log_bound_spreads[i1] = logsubexp(log(N1)+ compute_log_kernel(dist_LB_1, h, kernel),
                                       node_log_min_bounds[i1])
node_log_bound_spreads[i2] = logsubexp(log(N2)+ compute_log_kernel(dist_LB_2, h, kernel),
                                       node_log_min_bounds[i2])
```

I don't really find an unexpected result or something, just got confused when reading the code.
user: Charlie-XIAO
labels: Needs Investigation: Issue requires investigation
issue_27168: state: open
title: ENH HGBT histogram feature wise
body: #### Reference Issues/PRs
None

#### What does this implement/fix? Explain your changes.
This PR adds Cython histogram functions that calculate the histograms of 4 features at once. This way, the gradients and hessians can be used better. This increases performance in particular for single-threaded environments, see below.

#### Any other comments?
It is unclear to me, why the multi-threaded environments do not improve noticeably. This may be related to #14306.

#### benchmark
`python bench_hist_gradient_boosting_higgsboson.py --n-trees 100`
Note that this benchmark usually has quite some fluctuations in reported timings.
- Single threaded `export OMP_NUM_THREADS=1`
  - This PR
    ```
    Fit 100 trees in 153.223 s, (3100 total leaves)
    Time spent computing histograms: 84.157s
    Time spent finding best splits:  0.382s
    Time spent applying splits:      25.699s
    Time spent predicting:           3.624s
    fitted in 153.408s
    predicted in 24.846s, ROC AUC: 0.8228, ACC: 0.7415
    ```
  - MAIN
    ```
    Fit 100 trees in 172.712 s, (3100 total leaves)
    Time spent computing histograms: 105.948s
    Time spent finding best splits:  0.396s
    Time spent applying splits:      25.401s
    Time spent predicting:           3.687s
    fitted in 172.898s
    predicted in 25.716s, ROC AUC: 0.8228, ACC: 0.7415
    ```
- Multi-threaded (4 threads)
  - This PR
    ```
    Time spent computing histograms: 38.097s
    Time spent finding best splits:  0.233s
    Time spent applying splits:      8.211s
    Time spent predicting:           1.948s
    fitted in 66.434s
    ```
  - MAIN
    ```
    Time spent computing histograms: 38.640s
    Time spent finding best splits:  0.242s
    Time spent applying splits:      8.316s
    Time spent predicting:           1.986s
    fitted in 66.474s
    ```

user: lorentzenchr
issue_27159: state: open
title: RandomForest{Classifier,Regressor} split criterion documentation
body: ### Describe the issue linked to the documentation

There's no where in the documentation that explains what method is used to identify which values to consider as candidate splits. For example, for regression, an exhaustive method would be to sort each feature and then use the halfway point between each feature value. In this case there would be O(N*F) candidate splits where N=# data and F= # features. Another way would be to randomly choose K values between the min and max of each feature.

Does anyone know what method is used? Could someone point me to where in the code this occurs?

### Suggest a potential alternative/fix

_No response_
user: howarth
issue_27151: state: open
title: RFC remove some of our examples
body: TLDR: I think we have too many examples, in particular in
- [clustering](https://scikit-learn.org/stable/auto_examples/index.html#clustering)
- [ensemble](https://scikit-learn.org/stable/auto_examples/index.html#ensemble-methods)
- [generalized-linear-models](https://scikit-learn.org/stable/auto_examples/index.html#generalized-linear-models) This is the worst.
- [model-selection](https://scikit-learn.org/stable/auto_examples/index.html#model-selection)
- [nearest-neighbors](https://scikit-learn.org/stable/auto_examples/index.html#nearest-neighbors)
- [support-vector-machines](https://scikit-learn.org/stable/auto_examples/index.html#support-vector-machines)

On top, several examples don't render a plot in the gallery, e.g. [Release Highlights 1.0.0](https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_0_0.html#sphx-glr-auto-examples-release-highlights-plot-release-highlights-1-0-0-py).

I would like to discuss how we can improve this situation. My main worry is that, currently, a user does not find what she/he is looking for but gets overwhelmed by the sheer amount of examples.
user: lorentzenchr
issue_27145: state: open
title: DRAFT ENH Introduce `AVX` SIMD instructions for `ManhattanDistances.dist`
body: #### Reference Issues/PRs
A follow-up to https://github.com/scikit-learn/scikit-learn/issues/26010

#### What does this implement/fix? Explain your changes.
This PR demonstrates the minimal infrastructure required for proper building and runtime dispatch of `AVX` SIMD instructions. This means that scikit-learn can _optionally_ be built with `AVX` instructions. If, at runtime, `AVX` support is found on the runtime machine **and** scikit-learn was built with `AVX` support, then the optimized `AVX` instructions are leveraged when computing distances. Any other case results in the usual scalar loops currently found on `main`.

Concrete changes
- Introduction of `_simd/` directory housing new files
- Introduction of `simd.hpp` which provides a definition and extern declarations of templated Manhattan distance functions (`xsimd_manhattan_dist`)
- Introduction of `simd.cpp` which provides externed declarations of `xsimd_manhattan_dist`. This file serves as a translation unit to target with `-mavx` feature flags.
- Compiles the above into a composite `avx_dist_metrics` library which is linked against to isolate the rest of `_dist_metrics` from requiring additional feature flags (preventing auto-vectorization of unintended code, e.g. backup scalar loops)
- Provides templated scalar loops in C++ which match our current Cython implementaiton
- Adds `WITH_SIMD` macro to correctly compile SIMD optimized code, or default to a trivial implementation to satisfy compilation
- Adds `DIST_METRICS` macro to correctly allow inclusion of `_dist_optim.cpp` in `_dist_metrics.pyx` but **not** in other Cython files (necessary since any `cimport` of `_dist_metrics.pyx` forces the inclusion of `_dist_optim.cpp`)
- ~Adds build and runtime dependency on `archspec` to provide out-of-the-box support~ **Uses `xsimd` directly** for runtime microarchitecture feature detection (to determine whether building with `AVX` is possible, and whether running with `AVX` is possible) via `_runtime_check.pyx` and by compiling `runtime_check_program` in `setup.py` for a quick build-time check.
- Includes `xsimd` as a submodule (`~2MB` but could be reduced by half if we only include the `include` directory)
- Changes `self.{r}dist(...)` calls to `<Class>.rdist_csr(self, ...)` to work around Cython's idiosyncrasies when compiling to C++

TODO:
- [x] ~Resolve whether to add a dependency on `archspec` or vendor it~ Uses `xsimd` directly
- [ ] Resolve whether to add `xsimd` as a submodule, vendor an instance, or only include as build-time dependency for wheels and conda-forge

#### Any other comments?
`AVX` instructions were deployed in [late 2011 and 2012](https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX) and thus are likely found in a significant portion of our users' machines. Honestly, we may want to consider going to `AVX2` directly since it was released only a couple years later...

I don't know of any comprehensive surveys in the ML field which indicate the presence of these features on work machines and laptops. 

Initial benchmarks generated with [this script](https://gist.github.com/Micky774/5154eccb0450ea572d1720399b22a48e) (exported from a Jupyter notebook).
> [!NOTE]  
`PR_noruntime` is the case where the library is built with `AVX` support, but the runtime check fails to detect `AVX` capabilities. `PR_nobuild` is built **without** `AVX` capabilities.
`PR` is built with `AVX` capabilities, and uses them at runtime.

There are no slowdowns compared to main, and when built with `AVX` support, and it is detected at runtime, we see minor speedups even in cases of small `n_features`, to drastic speedups (`4-7x`) when `n_features` is large.
<details>
<summary> Plots </summary>

![e089e9cc-a6c4-4b62-a6ca-0d94e2babcaa](https://github.com/scikit-learn/scikit-learn/assets/34613774/2384b03c-2e52-4855-a4ee-6c6fec7c5a9d)

![ad00c343-0c0f-4438-8ea5-b5a98d1660ac](https://github.com/scikit-learn/scikit-learn/assets/34613774/dc4e22bf-f9d0-49cb-b027-819da738f2b6)

</details>
user: Micky774
issue_27135: state: open
title: API make naive Bayes API consistent in regards to "priors"
body: This PR intends to make the API of the class priors consistent across the different naive Bayes classifiers.

It deprecates `class_prior` and `fit_prior` in favor of a single parameter `priors` that is consistent with `GaussianNB` (introduced later) and `LinearDiscriminantAnalysis`.

I also introduce `class_log_prior_` in `GaussianNB` to be consistent with other estimators.

This change will also be helpful for the development of https://github.com/scikit-learn/scikit-learn/pull/22574. We can consistently set the `priors` attribute for all naive Bayes implementations and define a predictable API.
user: glemaitre
issue_27129: state: open
title: MNT Deprecate metrics.pairwise.paired_*_distances and paired_distances public functions
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Fixes #26982 

#### What does this implement/fix? Explain your changes.

Deprecates `metrics.pairwise.paired_*_distance functions` and `metrics.pairwise.paired_distance`

- [x] `metrics.pairwise.paired_euclidean_distances`
- [x] `metrics.pairwise.paired_manhattan_distances`
- [x] `metrics.pairwise.paired_cosine_distances`
- [x] `metrics.pairwise.paired_distances`

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Shreesha3112
labels: Waiting for Second Reviewer: First reviewer is done, need a second one!
issue_27117: state: open
title: Add sample_weight support to binning in HGBT
body: Use `sample_weight` in the binning of `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor`, or allow it via an option.

Currently, sample weights are ignored in the `_BinMapper`.

Some more context and history summarized by @NicolasHug [here](https://github.com/scikit-learn/scikit-learn/issues/24728#issuecomment-1288084267):
> I agree that it would make sense to support SW in the binner (although this is nuanced, see note below). Reading back the original PR, this was discussed extensively:
> 
>   - LightGBM implem doesn't take weights into account when binning [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-545138460)
> 
>   - I had a proposal to support SW in the binning: [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-556074060). Olivier and Andy seemed to be happy with it [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-581608597); there were some concerns from Adrin. To unblock the rest of the work, we were all happy to just not implement SW support in the Binner and leave that as potential future work.
> 
> 
> The estimators were still experimental at the time so we had more flexibility. Now that they're stable, bringing SW support in the Binner might require BC mitigations.
> 
> Also as a side note, [#15657 (comment)](https://github.com/scikit-learn/scikit-learn/issues/15657#issuecomment-556065553) may be relevant here: it's not super clear to me how we should handle SW in an estimator that performs some sort of subsambling during the training process (as is the case here during Binning).


user: lorentzenchr
labels: Needs Decision: Requires decision
issue_27114: state: open
title: Add processing pd.NA in OneHotEncoder.
body: Reference Issues/PRs
Fixes #26890

What does this implement/fix? Explain your changes.
Add processing of pd.NA for missing values.

Any other comments?
This is my PR. Maybe my code is not so good.


user: yanchick
issue_27113: state: open
title: WIP: Make standard scaler compatible to Array API
body: Here's my contribution from the EuroSciPy 2023 sprint. It's still work in progress and I won't have the time to continue the work before October. So if anyone else wants to take it from here, feel free to do so.

#### Reference Issues/PRs

See also #26024


#### What does this implement/fix? Explain your changes.

Make standard scaler compatible to Array API.


#### Any other comments?

Unfortunately, the current implementation breaks some unit tests of the standard scaler that are related to dtypes. That's because I wanted to make it work for torch.float16, but maybe that is not necessary and we should just support float32 and float64.

I'll also add some comments to the diff. See below.

user: AlexanderFabisch
issue_27109: state: open
title: Add baseline estimator to HGBT
body: ### Describe the workflow you want to enable

I would like to specify a baseline estimator like in `GradientBoostingRegressor(init=MyCoolBaselineEstimator_Maybe_a_linear_model)`.

### Describe your proposed solution

Add a parameter `baseline` (or if has to be the same, then `init`) to `HistGradientBoostingRegressor` and `HistGradientBoostingClassifier`.

### Describe alternatives you've considered, if relevant

The `StackingClassifier` does not do the same thing.

### Additional context

In particular a linear model as baseline could
1. Improve fit time
2. Allow for non-constant extrapolation on important features, quite in contrast to trees.
user: lorentzenchr
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_27108: state: open
title: DOC remove obsolete SVM example
body: Closes https://github.com/scikit-learn/scikit-learn/pull/26972

xref: https://github.com/scikit-learn/scikit-learn/pull/26972#discussion_r1282907863 and #27151

This example is pretty useless and outdated. Can simply be removed.

cc @glemaitre 
user: adrinjalali
issue_27096: state: open
title: ENH Add Array API compatibility tests for `*SearchCV` classes
body: RandomizedSearchCV and GridSearchCV appear to just work with Array API inputs.

This adds a test that makes sure that they will keep working.

For the common tests to pass we need `Ridge` to support the Array API.
user: betatim
issue_27090: state: open
title: TST Extend tests for `scipy.sparse.*array`
body: SciPy sparse matrices (i.e. `scipy.sparse.*matrix`) are tested but their sparse arrays counterpart (i.e. `scipy.sparse.*array`) aren't yet will become ubiquitous (see #26418).

Tests and their parameterizations (when they exist) must be adapted to include `scipy.sparse.*array` conditionally to versions of SciPy that support them (i.e. `scipy>=1.8`).

### Steps

:information_source: You can take https://github.com/scikit-learn/scikit-learn/pull/27095 as an example for your PRs.

#### 1. Choose one of the files to adapt

 - Indicate the file you are adapting (later referred to as `<filename>`) on this issue so no-one ends up duplicating work. E.g.

```
Hi, I am starting working on `<filename>`.
```

Please double check as this list might be outdated, but candidate files might be:

 - [x] https://github.com/scikit-learn/scikit-learn/pull/27107
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27095
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27093
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27099
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27100
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27121
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27116
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27097
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27128
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27131
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27133
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27143
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27144
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27148
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27150
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27132
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27101
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27160
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27162
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27163
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27171 
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27188
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27289
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27170
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27173
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27175
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27176
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27177
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27178
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27201
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27202
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27206
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27208
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27210
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27211
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27212
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27213
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27214
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27215
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27216
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27217
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27218
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27219
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27220
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27221
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27222
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27223
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27224
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27225
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27226
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27227
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27228
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27229
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27230
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27231
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27233
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27235
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27237
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27239
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27241
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27242
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27246
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27252
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27253
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27254
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27257
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27261
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27262
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27270
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27273
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27274
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27276
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27277
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27278
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27301
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27319
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27323
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27175
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27341
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27326
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27319
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27366
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27250
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27301
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27288
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27220
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27494
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27166
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27205
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27164
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27179
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27104
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27203
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27502
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27413
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27240
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27314
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27723
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27161
 - [x] https://github.com/scikit-learn/scikit-learn/pull/28101
 - [x] https://github.com/scikit-learn/scikit-learn/pull/27847


#### 2. Perform the following (non-exhaustive) changes for `<filename>`

 - Add test parametrization for tests using `*_matrix` (for instance see https://github.com/scikit-learn/scikit-learn/pull/27095/commits/eda7b16442869f6ccb37dd1b2a1b3bc47e4bec51)
 - Adapt existing test parametrization to test against sparse array (for instance see https://github.com/scikit-learn/scikit-learn/pull/27095/commits/6717f9903091e2780948aa9ef5765e2cd6d0f84a)
 - Extend or add test checking that type of sparse containers are preserve by functions. E.g. adding tests like (as proposed by https://github.com/scikit-learn/scikit-learn/issues/27090#issuecomment-1683849213):
	```python
    sparse_input = sparse_container(...)
    sparse_output = process_sparse_container(sparse_input)

    assert isinstance(sparse_output, type(sparse_input))
	```
 - Run tests for this file:
   ```
   # Run all the test in <filename>
   pytest <filename> -k test_to_run
   
   # Run a specific test in this file
   pytest <filename> -k test_to_run
   ```
 - If tests fail, the tested implementations has to be adapted (let us know if you need help)

#### 3. Create a pull request

 - Use this title:

```
TST Extend tests for `scipy.sparse.*array` in `<filename>`
```

 - Mention this issue in its description:

```
Towards #27090.
```

#### 4. Once the pull request is created

 - Ignore the error on the CI related the changelog: those PR need not update the changelog since nothing has changed for the users (until this issue is resolved), and maintainers must label PRs as "No Changelog Needed" for the check to be deactivated.

user: jjerphan
labels: module:test-suite: everything related to our tests
Meta-issue: General issue associated to an identified list of tasks
good first PR to review: Simple atomic PR to review
issue_27087: state: open
title: Add precompute in MultiTask Linear Models
body: ### Describe the workflow you want to enable

Adding a '**precompute**' parameter into constructor of **MultiTaskElasticNet**, **MultiTaskLasso**, **MultiTaskElasticNetCV**, and **MultiTaskLassoCV** in **linear_model/_coordinate_descent.py**
This parameter is only present in SingleTask models(**ElasticNet**, **Lasso**, **LinearModelCV**, **LassoCV**, **ElasticNetCV**), and has shown significant improvements in training speed.

### Describe your proposed solution

In **linear_model/_coordinate_descent.py**, a function **cd_fast.enet_coordinate_descent_multi_task()** in **enet_path()** will be executed when running these MultiTask models. However, this function does not have a precompute version in the **cd_fast.pyx**.

So the solution would be implementing a new function within **linear_model/_cd_fast.pyx**.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Is this because it's complicated or not feasible in mathematics?
user: kevinyin9
issue_27075: state: open
title: Numpy 2.0 release thoughts
body: Some notes from the EuroSciPy maintainer track about Numpy 2.0

You can track the progress Numpy is making towards 2.0 and what changes are coming in https://github.com/numpy/numpy/issues/24300.

Some things we should consider doing:

* [ ] pin to numpy<2 in the next releases to reduce the number of people who will install a version of scikit-learn that is incompatible with the numpy 2 version
* [ ] add some code in scikit-learn that detects that numpy 2 is installed and output a good error message for users
   * Olivier: do we really need this? Maybe the upper-bound pinning is enough.
* numpy 2 might come out at the end of 2023 or early 2024
* [ ] modify existing conda-forge packages to include a numpy<2 constraint
  * unclear if we can do this on PyPi as well
  * https://github.com/conda-forge/scikit-learn-feedstock/issues/227
* [x] once numpy 2.0 has been released we should build against that instead of "oldest-supported-numpy"
  * already do this now in the nightlies?
  * make sure that this works while still having 1.x.quite-old as the minimum required numpy
  * We no longer build against `oldest-supported-numpy` for Python 3.9 thanks to #27735 that was merged in `main` before the release of scikit-learn 1.4.

cc @ogrisel
user: betatim
issue_27047: state: open
title: Inconsistency in zero_division handling between precision/recall/f1 and precision_recall_curve/roc_curve related metrics
body: ### Describe the workflow you want to enable

The API offers the possibility to set the behavior upon stumbling upon a zero division issue when no positive label is present in the dataset, upon computing `precision_score`, `recall_score` or `f1_score` using the keyword argument:

    zero_division{“warn”, 0.0, 1.0, np.nan}, default=”warn”

    Sets the value to return when there is a zero division.

    Notes: - If set to “warn”, this acts like 0, but a warning is also raised. - If set to np.nan, such values will be excluded from the average.

    New in version 1.3: np.nan option was added.

`precision_recall_curve`, `roc_curve`, `roc_auc_score`, `average_precision_score `,  `label_ranking_average_precision_score`, despite having to compute precision or recall under the hood do not offer the same possibility.
While this is unlikely to pose problem in the micro averaging setting, it becomes more likely in the sample (AP and LRAP, possibly roc_auc_score)  or macro averaging setting.

For instance `precision_recall_curve` is not using the `precision_score` or `recall_score` functions:

https://github.com/scikit-learn/scikit-learn/blob/e4efd8b7961c1a16e862edc6c592e10dcd9d8697/sklearn/metrics/_ranking.py#L970-L985

In this implementation recall=1 and precision=0 when there is no positive example.

### Describe your proposed solution

Use the implemented `precision_score` and `recall_score` in all precision-recall and ROC curve functions. Add the same  `zero_division` kwarg and forward it to the `precision_score` and `recall_score` functions 

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I think this discussion is linked to:

- #24381 that is associated to an unmerged PR. My proposed solution would also enable to fix this issue in a more consistent way (by returning a positive `0.0` and a a zero division warning)
- #19085

Also to add more context (though changing this could break some existing code): `roc_curve` and `precision_recall_curve` do not handle this problem consistently:
```python

import numpy as np
import sklearn.metrics as skmet

y_true = np.zeros(10)
y_pred = np.random.uniform(size=y_true.shape)

skmet.roc_curve(y_true,y_pred,pos_label=1)
# UndefinedMetricWarning: No positive samples in y_true, true positive value should be meaningless
#  warnings.warn(
# Out: 
# (array([0. , 0.1, 1. ]),
# array([nan, nan, nan]),  # Recall or True positive Rate
# array([1.82341255, 0.82341255, 0.0795866 ]))

skmet.precision_recall_curve(y_true,y_pred,pos_label=1)
#UserWarning: No positive class found in y_true, recall is set to one for all thresholds.
#  warnings.warn(
#Out: 
#(array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]),  # Precision
# array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0.]),  # Recall or True positive Rate
# array([0.0795866 , 0.3813231 , 0.41105316, 0.56378517, 0.56951648,
#        0.60346455, 0.61754398, 0.61861517, 0.70285933, 0.82341255]))

# Compute AUC for the the PR curve
prec, recall, thresh = skmet.precision_recall_curve(y_true,y_pred,pos_label=1)
skmet
# UserWarning: No positive class found in y_true, recall is set to one for all thresholds.
skmet.auc(recall, prec)
# Out: 0.5
```
Which translates into an undefined AUC for ROC and a weird `0.5` value (nor 0 nor 1) for the AUC of the PR curve. I also find the warning confusing, it states that Recall is set to 1 for all thresholds, however the last value of the recall vector is `0.`...

On the same theme I think the ndcg_score is also somewhat inconsistent (ideal DCG is 0.0, hence a zero division somewhere, in the binary relevance case without positive example): 
```python
y_true = np.zeros((10,5))
y_pred = np.random.uniform(size=y_true_almnull.shape)

skmet.ndcg_score(y_true,y_pred)
Out: 0.0
```
If one wanted to perform sample averaging of the three different metrics (though for the PR curve AP and LRAP are the way to go to perform sample averaging), they would all behave differently upon encounter of a sample with zero positive example... 
user: qmarcou
labels: Needs Investigation: Issue requires investigation
issue_27045: state: open
title: Alternative way of estimating noise_variance in LASSOLarsIC in _least_angle.py
body: ### Describe the workflow you want to enable

I am working with LassoLarsIC, and the calculation of BIC/AIC in the algorithm. My main request is in the calculation of "self.noise_variance_". It uses the function estimate_noise_variance(self, X, y, positive), which takes as an input X and y, which are the original data. I understand why LinearRegression is used to calculate an estimate of the error and its variance. However, I think the estimate could also be calculated by using the variables with non-zero coefficients of lars_path, because the rest of the coefficients are essentially omitted from the regression. The way it is now, I think the degrees of freedom used in the calculation of "criterion" does not seem to align with the calculation of noise_variance_, because noise_variance_ estimate is the same for every coef_path regardless of selection. 

### Describe your proposed solution

```
if self.noise_variance is None and estimate_variance_with_lars = True: 
   noise_variance_ = np.zeros(coef_path_.shape[1], dtype=int)
   for i in coef_path_.shape[1]:
         noise_variance_[i] = self._estimate_noise_variance(
                            X[:, coef_path_[:, i]!=0], y, positive=self.positive
            )
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

In the following, LinearRegression has fit_intercept = False, but later the variance is estimated by using self.fit_intercept whose default is 1. It is a small issue, and I am not sure if it will affect the results, but just wanted to let you know.

```
        # X and y are already centered and we don't need to fit with an intercept
        ols_model = LinearRegression(positive=positive, fit_intercept=False)
        y_pred = ols_model.fit(X, y).predict(X)
        return np.sum((y - y_pred) ** 2) / (
            X.shape[0] - X.shape[1] - self.fit_intercept
        )
```
user: bakyapi
issue_27038: state: open
title: Document developer utils for parameter validation
body: ### Describe the issue linked to the documentation

https://github.com/scikit-learn/scikit-learn/pull/22722 and https://github.com/scikit-learn/scikit-learn/issues/24862 introduced parameter validation.

These tools should be documented under https://scikit-learn.org/dev/developers/utilities.html#validation-tools.

In addition, a lot of tests for raising errors in case of bad input seems to disappear. Are there common tests for it or is it still recommended to write tests a la
```python
def test_estim_raises():
    est = Estim(n_rounds="should be a number")
    with pytest.raises(ValueError):
        est.fit(X, y)
```

### Suggest a potential alternative/fix

_No response_
user: lorentzenchr
labels: Validation: related to input validation
issue_27035: state: open
title: HDBSCAN hierarchy tree plotting
body: ### Describe the workflow you want to enable

I have tried working with the newly introduces hdbscan clustering functionality and this works perfectly on the dataset I'm handling but I haven't found a way to plot the hierarchy tree of the clustering that clearly shows the effect of increasing the distance epsilon on the clustering. Although the implementation automatically clusters all the points, I think plotting the tree can really help with understanding the data and the clustering.
This is already part of the hdbscan package, but I can't find it in the sklearn implementation. 

### Describe your proposed solution

I would just add the functionalities from hdbscan in the same way as these work well. But it's better if it's part of the sklearn package.

### Describe alternatives you've considered, if relevant

Alternatively, sklearn could depend on hdbscan, but this is not the best solution.

### Additional context

_No response_
user: WEILMAX
issue_27028: state: open
title: Non-metric MDS gives nonsense results on the Digits dataset
body: ### Describe the bug

I am running metric and nonmetric MDS on the Digits dataset, and obtain nonsense results with `metric=False`. At the same time, my understanding is that non-metric MDS is more flexible and should not yield worse results compared to metric MDS. Is my understanding wrong, or does it suggest some problems with the non-metric MDS implementation?

### Steps/Code to Reproduce

```Python
%matplotlib notebook

import pylab as plt
import numpy as np
from sklearn.datasets import load_digits
from sklearn.manifold import MDS

digits = load_digits()
X, y = digits.data, digits.target

Z1 = MDS(n_components=2, n_init=1, random_state=42).fit_transform(X)
Z2 = MDS(n_components=2, n_init=1, random_state=42, metric=False).fit_transform(X)

titles = ['Metric MDS', 'Non-metric MDS']

fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(6, 3), layout='constrained')

for i,Z in enumerate([Z1, Z2]):
    ax = axs.ravel()[i]
    
    ax.set_aspect('equal', 'datalim')
    
    ind = np.random.permutation(X.shape[0])
    ax.scatter(*Z[ind].T, s=3, color=plt.cm.Dark2(y)[ind])

    ax.set_xticks([])
    ax.set_yticks([])
    for sp in ax.spines:
        ax.spines[sp].set_visible(False)
        
    ax.set_title(titles[i])
    
fig.savefig('digits-embed-mds-nonmetric.png', dpi=300)
```

### Expected Results

Non-metric MDS giving something sensible.

### Actual Results

![digits-embed-mds-nonmetric](https://github.com/scikit-learn/scikit-learn/assets/8970231/aa6af88a-75d0-4ef2-ab6e-6482c4df5ae8)


### Versions

```shell
System:
    python: 3.7.9 (default, Aug 31 2020, 12:42:55)  [GCC 7.3.0]
executable: /home/dmitry/anaconda3/bin/python
   machine: Linux-5.15.0-76-generic-x86_64-with-debian-bullseye-sid

Python dependencies:
          pip: 22.3.1
   setuptools: 65.6.3
      sklearn: 0.24.1
        numpy: 1.21.5
        scipy: 1.7.3
       Cython: 0.29.32
       pandas: 1.3.5
   matplotlib: 3.5.2
       joblib: 1.1.1
threadpoolctl: 2.2.0

Built with OpenMP: True
```

user: dkobak
labels: Needs Investigation: Issue requires investigation
issue_27023: state: open
title: Kernel density estimation on (semi-)bounded domains
body: ### Describe the workflow you want to enable

Kernel density estimates for bounded data are biased near the boundary because probability mass "spills out of the domain". It would be great to add a boundary correction to `KernelDensity`. For example, estimating the uniform density on the unit interval is severely biased near the boundaries.

```python
from matplotlib import pyplot as plt
import numpy as np
from sklearn.neighbors import KernelDensity


X = np.random.uniform(0, 1, (10_000, 1))
kde = KernelDensity(bandwidth="silverman").fit(X)
x = np.linspace(0, 1)
plt.plot(x, np.exp(kde.score_samples(x[:, None])))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/966348/dc213fc8-7475-4f82-82e3-d3b883ee44e7)


### Describe your proposed solution

Reflection methods reflect samples to be scored at the boundary such that the probability mass outside the support is "moved back" into the domain. Such a reflection could be implemented as follows.

1. Add a `bounds` parameter to the `KernelDensity` constructor with shape `(n_features, 2)` comprising the lower and upper bound in each dimension of the feature space.
2. Raise a `ValueError` in `fit` if one or more samples are outside the specified bounds.
3. Reflect samples to be scored at the boundaries in `score_samples` and combine contributions from all reflections using `logsumexp`.

Alternatively, a new estimator `BoundedKernelDensity` could inherit from `KernelDensity` to keep the `KernelDensity` API from ballooning. Would either be of interest for inclusion in scikit-learn?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: tillahoffmann
issue_27016: state: open
title: PoissonRegressor lbfgs solver giving coefficients of 0 and Runtime Warning
body: ### Describe the bug

See the following [stack exchange](https://stats.stackexchange.com/questions/622085/sklearn-poissonregressor-giving-all-coefficients-zero) post (the solution to my original issue was to use newton-cholesky solver)

When fitting a Poisson Regression (without regularization) to some dummy data I encounter:

- with lbfgs solver, a Runtime Warning, a non-zero intercept and all coefficients as zero
- with newton-cholesky solver, coefficients as expected

Some people on StackExchange have mentioned it is worth submitting an issue (there was a similar [one](https://github.com/scikit-learn/scikit-learn/issues/24752) faced with Logistic Regression).

### Steps/Code to Reproduce

```python
import statsmodels.api as sm
import statsmodels.formula.api as smf
import pandas as pd
from sklearn.linear_model import PoissonRegressor
from sklearn.preprocessing import (
    OneHotEncoder,
)
data = sm.datasets.get_rdataset('Insurance', package='MASS').data
# Fit Poisson regression using formula interface
formula = "Claims ~ C(District, Treatment(1)) + C(Group, Treatment('<1l')) + C(Age, Treatment('<25')) + Holders"
model_smf = smf.poisson(formula=formula, data=data).fit()
print(type(model_smf))
print(model_smf.summary())
# with sklearn OneHotEncoder

X_train_ohe = OneHotEncoder(sparse_output=False, drop=[1, "<1l", "<25"]).fit(data[["District", "Group", "Age"]])
X_train_ohe = pd.DataFrame(X_train_ohe.transform(data[["District", "Group", "Age"]]), columns=X_train_ohe.get_feature_names_out())

X_train = pd.concat([X_train_ohe, data[["Holders"]]], axis=1)
y_train = data["Claims"]

# one-hot encode the categorical columns, and drop the baseline column
# with lbfgs solver

model_sklearn_lbfgs = PoissonRegressor(alpha=0).fit(X_train, y_train)

print(model_sklearn_lbfgs.intercept_)
print(model_sklearn_lbfgs.coef_)
# with newton-cholesky solver

model_sklearn_nc = PoissonRegressor(alpha=0, solver='newton-cholesky').fit(X_train, y_train)

print(model_sklearn_nc.intercept_)
print(model_sklearn_nc.coef_)
```

### Expected Results

Expected intercept/coefficients (from statsmodels and sklearn with newton-cholesky solver):
```
2.8456859090224444
[-4.47436243e-01 -9.30629212e-01 -1.46547942e+00  1.00603354e+00
  4.71328611e-01 -5.98213625e-01  5.69685530e-01  6.85301286e-01
  2.22504699e+00 -1.58161172e-05]
```

### Actual Results

Result with lbfgs:
```
3.896592058397603
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
/home/akaash/Downloads/statsmodels-play/venv/lib/python3.10/site-packages/sklearn/linear_model/_linear_loss.py:290: RuntimeWarning: invalid value encountered in matmul
  grad[:n_features] = X.T @ grad_pointwise + l2_reg_strength * weights
```

### Versions

```shell
System:
    python: 3.10.9 (main, Mar  1 2023, 18:23:06) [GCC 11.2.0]
executable: /home/akaash/Downloads/statsmodels-play/venv/bin/python
   machine: Linux-5.19.0-46-generic-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.3.0
          pip: 23.2.1
   setuptools: 68.0.0
        numpy: 1.25.1
        scipy: 1.11.1
       Cython: None
       pandas: 2.0.3
   matplotlib: 3.7.2
       joblib: 1.3.1
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 6
         prefix: libopenblas
       filepath: /home/akaash/Downloads/statsmodels-play/venv/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
        version: 0.3.23
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 6
         prefix: libopenblas
       filepath: /home/akaash/Downloads/statsmodels-play/venv/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 6
         prefix: libgomp
       filepath: /home/akaash/Downloads/statsmodels-play/venv/lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

user: akaashp2000
issue_27014: state: open
title: Pipeline throws TypeError on stateless transformers
body: ### Describe the bug

In PR #26952 @thomasjpfan was hinting that transformers in a pipeline are allowed to be stateless. They are, but only if no other step of the pipeline implements a `fit()`.

However, if one of the steps implements a `fit`, the previous steps are expected to be stateful transformers, too. This is because `def _validate_steps()` (which checks the methods of all the steps in the pipeline) only checks the steps if `.fit()` is called.

Should `_validate_steps() ` be modified to be less strict about stateless transformers? And should it then also be run on `transform()`, and all the other possible methods?

### Steps/Code to Reproduce

```
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LinearRegression
import numpy as np

class DoubleIt:

    def transform(self, X, y=None):
        return 2*X

X = np.array([[1, 2, 3], [4, 5, 6]])
p = Pipeline([
            ('double1', DoubleIt()),
            ('double2', DoubleIt()),
            ('linreg', LinearRegression())     # same results with ('linreg', None), but not with "passthrough"
            ])

p.fit(X)
```

### Expected Results

No error is raised.

### Actual Results

```
Traceback (most recent call last):
  File "/home/stefanie/Python/scikit-learn_dev/scikit-learn/::::::::::::::::::::::::::::.py", line 126, in <module>
    p.fit(X)
  File "/home/stefanie/Python/scikit-learn_dev/scikit-learn/sklearn/base.py", line 1215, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/stefanie/Python/scikit-learn_dev/scikit-learn/sklearn/pipeline.py", line 456, in fit
    Xt = self._fit(X, y, routed_params)
  File "/home/stefanie/Python/scikit-learn_dev/scikit-learn/sklearn/pipeline.py", line 372, in _fit
    self._validate_steps()
  File "/home/stefanie/Python/scikit-learn_dev/scikit-learn/sklearn/pipeline.py", line 242, in _validate_steps
    raise TypeError(
TypeError: All intermediate steps should be transformers and implement fit and transform or be the string 'passthrough' '<__main__.DoubleIt object at 0x7fd4b028f310>' (type <class '__main__.DoubleIt'>) doesn't
```

### Versions

```shell
System:
    python: 3.10.6 (main, Oct 10 2022, 12:43:33) [GCC 9.4.0]
executable: /home/stefanie/.pyenv/versions/3.10.6/envs/scikit-learn_dev/bin/python
   machine: Linux-5.15.0-78-generic-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.4.dev0
          pip: 23.2.1
   setuptools: 63.2.0
        numpy: 1.24.3
        scipy: 1.10.1
       Cython: 0.29.34
       pandas: 2.0.1
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/stefanie/.pyenv/versions/3.10.6/envs/scikit-learn_dev/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so
        version: 0.3.21
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/stefanie/.pyenv/versions/3.10.6/envs/scikit-learn_dev/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 8

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
        version: None
    num_threads: 8
```

user: StefanieSenger
issue_27012: state: open
title: inverse_transform of SimpleImputer with empty features changes order of columns
body: ### Describe the bug

When one uses a SimpleImputer with `keep_empty_features=False` and `add_indicator=True`, then if a column has only missing values during fit, but has values during transform, then the inverse transform will shift all columns to the right of that column one to the left. 

(Note that I found this while experimenting and is not something that we rely on being fixed)

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.impute import SimpleImputer

X1 = np.array(
    [
        [np.nan, 2.0, 3.0],
        [np.nan, 2.0, 3.0],
    ]
)

X2 = np.array(
    [
        [1.0, 2.0, 3.0],
        [1.0, 2.0, 3.0],
    ]
)

imputer = SimpleImputer(add_indicator=True)
imputer.fit(X1)
print(imputer.inverse_transform(imputer.transform(X2)))
```

### Expected Results

I think the best way to go is to fill the missing columns with the value from `missing_values`.

```python
[[nan 2. 3.]
 [nan 2. 3.]]
```

### Actual Results

```python
[[2. 3. 0.]
 [2. 3. 0.]]
```

### Versions

```shell
System:
    python: 3.11.4
executable: [redacted]
   machine: macOS-13.4.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.3.0
          pip: 23.2
   setuptools: 68.0.0
        numpy: 1.25.1
        scipy: 1.11.1
       Cython: None
       pandas: 2.0.3
   matplotlib: 3.7.2
       joblib: 1.3.0
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libopenblas
       filepath: [redacted]
        version: 0.3.23
threading_layer: openmp
   architecture: VORTEX

       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: libomp
       filepath: [redacted]
        version: None
```

user: JoelHuberQC
issue_27004: state: open
title: Upload musllinux wheel to PyPI
body: ### Describe the workflow you want to enable

I want to pip install scikit-learn==1.3.0 into a docker alpine 3.11 Installation, but it appears that here are no wheels available for that platform.

Pip shows my my platform tags as
```
  cp311-cp311-musllinux_1_2_x86_64
  cp311-cp311-musllinux_1_1_x86_64
  cp311-cp311-musllinux_1_0_x86_64
  cp311-cp311-linux_x86_64
  cp311-abi3-musllinux_1_2_x86_64
  cp311-abi3-musllinux_1_1_x86_64
  cp311-abi3-musllinux_1_0_x86_64
  cp311-abi3-linux_x86_64
  cp311-none-musllinux_1_2_x86_64
  cp311-none-musllinux_1_1_x86_64
  cp311-none-musllinux_1_0_x86_64
  cp311-none-linux_x86_64
```

Available wheels in pypy are
```
scikit_learn-1.3.0-cp311-cp311-win_amd64
scikit_learn-1.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64
scikit_learn-1.3.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64
scikit_learn-1.3.0-cp311-cp311-macosx_12_0_arm64.whl
scikit_learn-1.3.0-cp311-cp311-macosx_10_9_x86_64.whl
```



### Describe your proposed solution

I unfortunately dont understand enough about wheels, but Can I build a wheel for my platform or use "scikit_learn-1.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64" somehow?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: mapa17
labels: Needs Triage: Issue requires triage
issue_27003: state: open
title: Add concept of "soft deprecation" to development tools
body: #### What does this implement/fix? Explain your changes.
This adds the concept of "soft deprecation" from https://peps.python.org/pep-0387/#soft-deprecation.

The idea of this PR is to find out what other core devs think of this concept and idea.

TL;DR: is used in core Python to indicate that part of the API should not be used for new code, but won't be removed (so you can keep using it in existing code).

When I saw this I thought it could be an interesting tool to use to deal with having "too much stuff" in scikit-learn. Two examples for "soft deprecation" are the MLP code and the Passive Aggressive Classifier.

Questions:
* is this useful? Too complicated? Impractical?
* how do we deal with the text being taken verbatim from the PEP?
user: betatim
issue_27000: state: open
title: make cross_val_predict use cross_validate
body: Right now `cross_val_score` is a wrapper for `cross_validate`, but `cross_val_predict` has its own implementation.

We could make it also a wrapper around `cross_validate`, but for that to happen, we need to make `cross_validate` also return predictions. For that, we would also need to be able to pass `y_pred, y_true` to scorers instead of passing the estimator, which would also remove the need to cache in `_MultimetricScorer` and make things quite a bit cleaner.

cc @glemaitre 

xref: https://github.com/scikit-learn/scikit-learn/pull/26896#pullrequestreview-1560866760
user: adrinjalali
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_26999: state: open
title: `ValueError: Input contains NaN.` in `sklearn.manifold.smacof`
body: ### Describe the bug

I accidentally stumbled onto a `ValueError` when executing `smacof`. I hacked into `_mds.py` to save both the offending `dissimilarities` as well as the randomly generated `X`, then cut them down to minimal shape that still exhibits the error. This data is attached below in the MCVE.

### Steps/Code to Reproduce

```
import numpy as np                          
import sklearn.manifold                     
                                            
dis = np.array([
    [0.0, 1.732050807568877, 1.7320508075688772], 
    [1.732050807568877, 0.0, 6.661338147750939e-16],
    [1.7320508075688772, 6.661338147750939e-16, 0.0]
])  
init = np.array([
    [0.08665881585055124, 0.7939114643387546],
    [0.9959834154297658, 0.7555546025640025],
    [0.8766008278401566, 0.4227358815811242]
])  
sklearn.manifold.smacof(dis, init=init, normalized_stress="auto", metric=False, n_init=1)
```

### Expected Results

No errors

### Actual Results

```
Traceback (most recent call last):
  File ".../rep_error.py", line 14, in <module>
    sklearn.manifold.smacof(dis, init=init, normalized_stress="auto", metric=False, n_init=1)
  File ".../.direnv/python-3.9.5/lib/python3.9/site-packages/sklearn/manifold/_mds.py", line 329, in smacof
    pos, stress, n_iter_ = _smacof_single(
  File ".../.direnv/python-3.9.5/lib/python3.9/site-packages/sklearn/manifold/_mds.py", line 128, in _smacof_single
    dis = euclidean_distances(X)
  File ".../.direnv/python-3.9.5/lib/python3.9/site-packages/sklearn/metrics/pairwise.py", line 310, in euclidean_distances
    X, Y = check_pairwise_arrays(X, Y)
  File ".../.direnv/python-3.9.5/lib/python3.9/site-packages/sklearn/metrics/pairwise.py", line 156, in check_pairwise_arrays
    X = Y = check_array(
  File ".../.direnv/python-3.9.5/lib/python3.9/site-packages/sklearn/utils/validation.py", line 959, in check_array
    _assert_all_finite(
  File ".../.direnv/python-3.9.5/lib/python3.9/site-packages/sklearn/utils/validation.py", line 124, in _assert_all_finite
    _assert_all_finite_element_wise(
  File ".../.direnv/python-3.9.5/lib/python3.9/site-packages/sklearn/utils/validation.py", line 173, in _assert_all_finite_element_wise
    raise ValueError(msg_err)
ValueError: Input contains NaN.
```

### Versions

```shell
System:
    python: 3.9.5 (default, Nov 23 2021, 15:27:38)  [GCC 9.3.0]
executable: .../.direnv/python-3.9.5/bin/python
   machine: Linux-5.4.0-148-generic-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.3.0
          pip: 23.2.1
   setuptools: 44.0.0
        numpy: 1.25.1
        scipy: 1.11.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.1
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 32
         prefix: libgomp
       filepath: .../.direnv/python-3.9.5/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: .../.direnv/python-3.9.5/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
        version: 0.3.23
threading_layer: pthreads
   architecture: Zen

       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: .../.direnv/python-3.9.5/lib/python3.9/site-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Zen
```

user: amadanmath
labels: Needs Investigation: Issue requires investigation
issue_26983: state: open
title: FEA Introduce `PairwiseDistances`, a generic back-end for `pairwise_distances`
body: #### Reference Issues/PRs
Towards https://github.com/scikit-learn/scikit-learn/pull/23958
Supersedes https://github.com/scikit-learn/scikit-learn/pull/25561

#### What does this implement/fix? Explain your changes.
From #25561:
 This simplifies the original implementation of `PairwiseDistance` by @jjerphan, with the following differences:
- `PairwiseDistance{32,64}` doesn't subclass `BaseDistancesReduction{32,64}` anymore.
- This allows to add `_parallel_on_{X,Y}` methods to `PairwiseDistance{32,64}`, since these methods are decorated with `@final` in `BaseBaseDistancesReduction{32,64}` and thus can't be overwritten.
- This also remove the chunk computing mechanism, by considering only the case `chunk_size = 1`, as proposed by @ogrisel in [this comment](https://github.com/scikit-learn/scikit-learn/pull/23958#issuecomment-1280517094).
- This doesn't implement the Euclidean specialization yet to make benchmarks simpler.

This PR:
- Preserves existing specializations and dispatches based on benchmark-driven heuristic (benchmarks will be published soon)
- Simplifies usability criteria for `PairwiseDistances`
- Changes backend dispatch process in `pairwise_distances`
- Adds `n_jobs` parameter to `PairwiseDistances` to preserve `n_jobs` semantics from `pairwise_distances`
- Moves `X_is_Y` attribute to `PairwiseDistances` to minimize change to unrelated API (all that would be affected by `DatasetsPair`)
- Cleans misc. changes/comments and updates PR
- Minimizes diff

#### Any other comments?
Benchmarks will be coming soon
user: Micky774
issue_26982: state: open
title: API Deprecate `paired_distances` and `paired_*_distances`
body: ## Background

We have several public functions `paired_*_distances` which exist primarily as specialized implementations for the public `paired_distances` function. This function is thus either a wrapper around the specialized `paired_*_distances` or a simple iteration over the input data. During the drafting meeting, we discussed the deprecation of these functions. This discussion continued on a recent PR ([start here](https://github.com/scikit-learn/scikit-learn/pull/22705#issuecomment-1655985340)). I wanted to open this to see if we could come to a quick consensus on our intent for the `paired_*` functions.

Personally, I definitely want to see `paired_*_distances` deprecated. I have no idea how widely used `paired_distances` is in practice. I believe @thomasjpfan has a better sense of this after some github searches. If it is not too widely used, I am also in favor of deprecating it as a whole.

As @adrinjalali mentioned ([cf. comment](https://github.com/scikit-learn/scikit-learn/pull/22705#issuecomment-1660010966)) the `paired_distances` function is a really light function that most users ought to be able to write themselves. It doesn't really make sense for us to provide it for them...

cc: @scikit-learn/core-devs 

**Edit: Consensus is to deprecate both `paired_*_distances` and `paired_distances`**
user: Micky774
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_26963: state: open
title: Path to adopt 32-bit implementations for `{KD, Ball}Tree`
body: ## Motivation
Having and using 32-bit implementations of `{KD, Ball}Tree` allows for better preservation of dtype, lower memory footprint, and more consistent Cython code.

## Strategy
Work has already been started in #25914 to add the code for the new 32-bit implementations. These will not be directly used yet, and the PR instead focuses on the actual creation of the new classes. Consequently, `{KD, Ball}Tree` are bound to `{KD, Ball}Tree64` for consistency and backwards compatibility.

Following this PR, we will need to begin an API deprecation to move users away from constructing trees directly, and instead using a factory method (similar to `DistanceMetric.get_metric`). Then, later, we can separate `{KD, Ball}Tree` from `{KD, Ball}Tree64` so that we have a singular dispatcher `{KD, Ball}Tree` and two type-specialized implementations. 

The deprecation process involves:
1. Introducing `get_tree(...)` to `{KD, Ball}Tree64` with the same signature of `BinaryTree.__init__`, which is in charge of constructing the type-specialized trees directly.
2. Adding a `FutureWarning` to `BinaryTree.__init__` to begin deprecation, and suppressing the warning using a context manager in `{KD, Ball}Tree64.get_tree` to enforce it as the "correct" way to construct the trees.
3. **Complete the deprecation** by introducing a separate `{KD, Ball}Tree` into the hierarchy which comes with the same `get_tree` method and fully remove the `FutureWarning` from `{KD, Ball}Tree{32, 64}.__init__` since they should no longer be directly constructed, as well as getting rid of the `{KD, Ball}Tree{32, 64}.get_tree` method since it should only exist in `{KD, Ball}Tree`.

At the end, the expected API is:
```python
from sklearn.neighbors import KDTree

# Data
X = ...

# tree is thus an instance of KDTree{32, 64} based on X.dtype
tree = KDTree.get_tree(X, ...)
```

cc: @thomasjpfan @jjerphan @OmarManzoor 
user: Micky774
labels: Needs Decision - API: Requires decision regarding API
issue_26962: state: open
title: DOC Add link to plot_tree_regression.py example
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
This pr adds link in [DecisionTreeRegressor](http://localhost:63342/scikit-learn/doc/_build/html/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html#sklearn.tree.DecisionTreeRegressor) to example file `tree/plot_tree_regression.py` as mentioned in #26927 
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Tialo
issue_26961: state: open
title: Agglomerative clustering training error for seuclidean/mahalanobis affinity and single linkage
body: ### Describe the bug

When trying Agglomerative clustering model training with the affinity as 'seuclidean' or 'mahalanobis' and the linkage as 'single' the training fails. The same affinity values along with other linkage such as 'average' options executes for model training. There's no specification given for this issue. Also, in the code I can see the handling for the single linkage is different and there is some cython code which is not accessible.

### Steps/Code to Reproduce

```

from sklearn.cluster import AgglomerativeClustering
from sklearn.datasets import load_iris
model = AgglomerativeClustering(affinity='mahalanobis', linkage='average')
data = load_iris(as_frame=True)['data']
model.fit(data)

```

### Expected Results

No error should be thrown sig

### Actual Results

![image](https://github.com/scikit-learn/scikit-learn/assets/69189799/c3da5d93-b215-4fd8-8610-a4c8ce632f8d)


### Versions

```shell
System:
    python: 3.8.17 (default, Jul  5 2023, 21:04:15)  [GCC 11.2.0]
executable: /home/albint/miniconda3/envs/myenv/bin/python
   machine: Linux-5.15.0-78-generic-x86_64-with-glibc2.17

Python dependencies:
      sklearn: 1.3.0
          pip: 23.1.2
   setuptools: 67.8.0
        numpy: 1.24.4
        scipy: 1.10.1
       Cython: 3.0.0
       pandas: 2.0.3
   matplotlib: 3.7.2
       joblib: 1.3.1
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /home/albint/miniconda3/envs/myenv/lib/python3.8/site-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so
        version: 0.3.21
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /home/albint/miniconda3/envs/myenv/lib/python3.8/site-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libgomp
       filepath: /home/albint/miniconda3/envs/myenv/lib/python3.8/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

user: albin02t
issue_26955: state: open
title: Make cross_val_score accept multi-metric scoring
body: I'm not sure why we deliberately exclude multi-metric scoring in `cross_val_score`. `cross_validate` (which `cross_val_score` calls, supports it, and we could simply return the multi-metric scores calculated by it.

This is our `cross_val_score`:

https://github.com/scikit-learn/scikit-learn/blob/405a5a09503a7f2cd241f8a5b1a1d2368dcc2676/sklearn/model_selection/_validation.py#L576-L591
user: adrinjalali
labels: Needs Decision - API: Requires decision regarding API
issue_26951: state: open
title: DOC Add links to plot_document_clustering example
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs

This adds links to `text/plot_document_clustering.py` examples as mentioned in https://github.com/scikit-learn/scikit-learn/issues/26927
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
This PR adds links to `text/plot_document_clustering.py` example file to classes:
1. `TfidfVectorizer`
2. `HashingVectorizer`

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Tialo
labels: Waiting for Second Reviewer: First reviewer is done, need a second one!
issue_26950: state: open
title: DOC Notebook style and enhanced descriptions and add example links for feature_selection.RFE
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
issue [https://github.com/scikit-learn/scikit-learn/issues/26927](https://github.com/scikit-learn/scikit-learn/issues/26927)


#### What does this implement/fix? Explain your changes.

This PR adds example links for `feature_selection.RFE`

`Class`: [feature_selection.RFE](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html#sklearn.feature_selection.RFE)

Related Example files:

* [sphx-glr-auto-examples-feature-selection-plot-rfe-digits-py](https://scikit-learn.org/stable/auto_examples/feature_selection/plot_rfe_digits.html#sphx-glr-auto-examples-feature-selection-plot-rfe-digits-py)

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Shreesha3112
labels: good first issue: Easy with clear instructions to resolve
issue_26945: state: open
title: [PLSRegression] The standard scikit interface should have the argument in lowercase "y" not uppercase "Y"
body: ### Describe the bug

The standard scikit interface should have the argument in lowercase "y" not uppercase "Y". Would you please modify it?

[[https://github.com/scikit-learn/scikit-learn/blob/7f9bad99d6e0a3e8ddf92a7e5561245224dab102/sklearn/cross_decomposition/_pls.py#L622](https://urldefense.com/v3/__https:/github.com/scikit-learn/scikit-learn/blob/7f9bad99d6e0a3e8ddf92a7e5561245224dab102/sklearn/cross_decomposition/_pls.py*L622__;Iw!!OzAIPA!HpZIZynDig4VRjHXL6okypqqpe44CBO0ltsalU3v7vfpWdUqghoJSzun4kIJK72EuyflVXKZZEcLic5jD0n-KvEP$)](https://github.com/scikit-learn/scikit-learn/blob/7f9bad99d6e0a3e8ddf92a7e5561245224dab102/sklearn/cross_decomposition/_pls.py#L622)

### Steps/Code to Reproduce

Please see the above link.

### Expected Results

Change the  lowercase "y" instead of uppercase "Y".

### Actual Results

 lowercase "y"

### Versions

```shell
Scikit-learn 1.3.0
```

user: m-kaka
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_26941: state: open
title: Make sure `set_output` is not rendered by sphinx when disabled
body: xref: https://github.com/scikit-learn/scikit-learn/pull/26940#issue-1827735779

When `auto_wrap_output_keys=None`, there shouldn't be any documented `set_output` method since it doesn't exist.

cc @thomasjpfan 
user: adrinjalali
issue_26927: state: open
title: Add links to examples from the docstrings and user guides
body: We have a rich set of examples covering a very broad range of issues, but they're not necessarily easily discoverable via the section on the bottom of the API pages.

This meta-issue is to keep track of the examples for which we've already included links in the docs, and to fix the rest.

An example of how to include links to examples in API docs and the user guide can be found here: https://github.com/scikit-learn/scikit-learn/pull/26926 . Please see how it can be done before claiming an example.

For each example, please check if it's already included in the docs, either user guides or API docs, and if not, claim it by leaving a comment at the bottom of this issue and open a pull request for it. You can open one PR for more than one example if they're very related.

Note that not all examples need to be included in this project. For instance, if an example is only a usage example, and putting a link to it doesn't add much for the users, then we can ignore it (please do leave a comment if this is your finding). Also, not all examples are in a good shape, and in your PR you can also improve the example itself.

**How to create a link to an example?**
To create a link to the example `examples/developing_estimators/sklearn_is_fitted.py`, you can write:

```
:ref:`sphx_glr_auto_examples_developing_estimators_sklearn_is_fitted.py`
```
which would automatically create the link. Notice the pattern where slashes (`/`) are converted to an underscore (`_`) and the `sphx_glr_auto_` prefix.

__IMPORTANT__: Please limit your pull requests to a limited scope, single or two classes, or a single example so that it can be reviewed easily.

Here's a list of all examples:

- examples/applications:
  - [ ] plot_cyclical_feature_engineering.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26971
  - [ ] plot_digits_denoising.py
  - [ ] plot_face_recognition.py
  - [ ] plot_model_complexity_influence.py
  - [ ] plot_outlier_detection_wine.py
  - [ ] plot_out_of_core_classification.py
  - [ ] plot_prediction_latency.py
  - [ ] plot_species_distribution_modeling.py
  - [ ] plot_stock_market.py
  - [ ] plot_tomography_l1_reconstruction.py
  - [ ] plot_topics_extraction_with_nmf_lda.py
  - [ ] wikipedia_principal_eigenvector.py

- examples/bicluster:
  - [ ] plot_bicluster_newsgroups.py
  - [ ] plot_spectral_biclustering.py
  - [ ] plot_spectral_coclustering.py

- examples/calibration:
  - [ ] plot_calibration_curve.py
  - [ ] plot_calibration_multiclass.py
  - [ ] plot_calibration.py
  - [ ] plot_compare_calibration.py

- examples/classification:
  - [ ] plot_classification_probability.py
  - [ ] plot_classifier_comparison.py
  - [ ] plot_digits_classification.py
  - [ ] plot_lda.py
  - [ ] plot_lda_qda.py

- examples/cluster:
  - [ ] plot_adjusted_for_chance_measures.py
  - [ ] plot_affinity_propagation.py
  - [ ] plot_agglomerative_clustering_metrics.py
  - [ ] plot_agglomerative_clustering.py
  - [ ] plot_agglomerative_dendrogram.py
  - [ ] plot_birch_vs_minibatchkmeans.py
  - [ ] plot_bisect_kmeans.py
  - [ ] plot_cluster_comparison.py
  - [ ] plot_cluster_iris.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27799
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26974
  - [ ] plot_coin_segmentation.py
  - [ ] plot_coin_ward_segmentation.py
  - [ ] plot_color_quantization.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27799
  - [ ] plot_dbscan.py
  - [ ] plot_dict_face_patches.py
  - [ ] plot_digits_agglomeration.py
  - [ ] plot_digits_linkage.py
  - [ ] plot_face_compress.py
  - [ ] plot_feature_agglomeration_vs_univariate_selection.py
  - [ ] plot_hdbscan.py
  - [ ] plot_inductive_clustering.py
  - [ ] plot_kmeans_assumptions.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27799
  - [ ] plot_kmeans_digits.py
      - [ ] https://github.com/scikit-learn/scikit-learn/pull/26981
      - [ ] https://github.com/scikit-learn/scikit-learn/pull/27799
  - [ ] plot_kmeans_plusplus.py
  - [ ] plot_kmeans_silhouette_analysis.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27799
  - [ ] plot_kmeans_stability_low_dim_dense.py
  - [ ] plot_linkage_comparison.py
  - [ ] plot_mean_shift.py
  - [ ] plot_mini_batch_kmeans.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27799
  - [ ] plot_optics.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26970
  - [ ] plot_segmentation_toy.py
  - [ ] plot_ward_structured_vs_unstructured.py

- examples/compose:
  - [ ] plot_column_transformer_mixed_types.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26904
  - [ ] plot_column_transformer.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26904
  - [ ] plot_compare_reduction.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26904
  - [ ] plot_digits_pipe.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26904
  - [ ] plot_feature_union.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26904
  - [ ] plot_transformed_target.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26904

- examples/covariance:
  - [ ] plot_covariance_estimation.py
  - [ ] plot_lw_vs_oas.py
  - [ ] plot_mahalanobis_distances.py
  - [ ] plot_robust_vs_empirical_covariance.py
  - [ ] plot_sparse_cov.py

- examples/cross_decomposition:
  - [ ] plot_compare_cross_decomposition.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26966
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26934
  - [ ] plot_pcr_vs_pls.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26934

- examples/datasets:
  - [ ] plot_digits_last_image.py
  - [ ] plot_iris_dataset.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26973
  - [ ] plot_random_dataset.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27056
  - [ ] plot_random_multilabel_dataset.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27517

- examples/decomposition:
  - [ ] plot_faces_decomposition.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26932
  - [ ] plot_ica_blind_source_separation.py
  - [ ] plot_ica_vs_pca.py
  - [ ] plot_image_denoising.py
  - [ ] plot_incremental_pca.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26932
  - [ ] plot_kernel_pca.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26932
  - [ ] plot_pca_3d.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26932
  - [ ] plot_pca_iris.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26932
  - [ ] plot_pca_vs_fa_model_selection.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26932
  - [ ] plot_pca_vs_lda.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26932
  - [ ] plot_sparse_coding.py
  - [ ] plot_varimax_fa.py

- examples/developing_estimators:
  - [ ] sklearn_is_fitted.py: https://github.com/scikit-learn/scikit-learn/pull/26926

- examples/ensemble:
  - [ ] plot_adaboost_hastie_10_2.py
  - [ ] plot_adaboost_multiclass.py
  - [ ] plot_adaboost_regression.py
  - [ ] plot_adaboost_twoclass.py
  - [ ] plot_bias_variance.py
  - [ ] plot_ensemble_oob.py
  - [ ] plot_feature_transformation.py
  - [ ] plot_forest_hist_grad_boosting_comparison.py
  - [ ] plot_forest_importances_faces.py
  - [ ] plot_forest_importances.py
  - [ ] plot_forest_iris.py
  - [ ] plot_gradient_boosting_categorical.py
  - [ ] plot_gradient_boosting_early_stopping.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27025
  - [ ] plot_gradient_boosting_oob.py
  - [ ] plot_gradient_boosting_quantile.py
  - [ ] plot_gradient_boosting_regression.py
  - [ ] plot_gradient_boosting_regularization.py
  - [ ] plot_isolation_forest.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26975
  - [ ] plot_monotonic_constraints.py
  - [ ] plot_random_forest_embedding.py
  - [ ] plot_random_forest_regression_multioutput.py
  - [ ] plot_stack_predictors.py
  - [ ] plot_voting_decision_regions.py
  - [ ] plot_voting_probas.py
  - [ ] plot_voting_regressor.py

- examples/exercises:
  - [ ] plot_cv_diabetes.py
  - [ ] plot_cv_digits.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/28073
  - [ ] plot_digits_classification_exercise.py
  - [ ] plot_iris_exercise.py

- examples/feature_selection:
  - [ ] plot_feature_selection_pipeline.py
  - [ ] plot_feature_selection.py
  - [ ] plot_f_test_vs_mi.py
  - [ ] plot_rfe_digits.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26950
  - [ ] plot_rfe_with_cross_validation.py
  - [ ] plot_select_from_model_diabetes.py

- examples/gaussian_process:
  - [ ] plot_compare_gpr_krr.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26980
  - [ ] plot_gpc_iris.py
  - [ ] plot_gpc_isoprobability.py
  - [ ] plot_gpc.py
  - [ ] plot_gpc_xor.py
  - [ ] plot_gpr_co2.py
  - [ ] plot_gpr_noisy.py
  - [ ] plot_gpr_noisy_targets.py
  - [ ] plot_gpr_on_structured_data.py
  - [ ] plot_gpr_prior_posterior.py

- examples/impute:
  - [ ] plot_iterative_imputer_variants_comparison.py
	  - [ ] #27053
  - [ ] plot_missing_values.py
	  - [ ] #27053

- examples/inspection:
  - [ ] plot_causal_interpretation.py
  - [ ] plot_linear_model_coefficient_interpretation.py
  - [ ] plot_partial_dependence.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27156
  - [ ] plot_permutation_importance_multicollinear.py
  - [ ] plot_permutation_importance.py

- examples/kernel_approximation:
  - [ ] plot_scalable_poly_kernels.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27527

- examples/linear_model:
  - [ ] plot_ard.py
  - [ ] plot_bayesian_ridge_curvefit.py
  - [ ] plot_elastic_net_precomputed_gram_matrix_with_weighted_samples.py
  - [ ] plot_huber_vs_ridge.py
  - [ ] plot_iris_logistic.py
  - [ ] plot_lasso_and_elasticnet.py
  - [ ] plot_lasso_coordinate_descent_path.py
  - [ ] plot_lasso_dense_vs_sparse_data.py
  - [ ] plot_lasso_lars_ic.py
  - [ ] plot_lasso_lars.py
  - [ ] plot_lasso_model_selection.py
  - [ ] plot_logistic_l1_l2_sparsity.py
  - [ ] plot_logistic_multinomial.py
  - [ ] plot_logistic_path.py
  - [ ] plot_logistic.py
  - [ ] plot_multi_task_lasso_support.py
  - [ ] plot_nnls.py
  - [ ] plot_ols_3d.py
  - [ ] plot_ols.py
  - [ ] plot_ols_ridge_variance.py
  - [ ] plot_omp.py
  - [ ] plot_poisson_regression_non_normal_loss.py
  - [ ] plot_polynomial_interpolation.py
  - [ ] plot_quantile_regression.py
  - [ ] plot_ransac.py
  - [ ] plot_ridge_coeffs.py
  - [ ] plot_ridge_path.py
  - [ ] plot_robust_fit.py
  - [ ] plot_sgd_comparison.py
  - [ ] plot_sgd_early_stopping.py
  - [ ] plot_sgd_iris.py
  - [ ] plot_sgd_loss_functions.py
  - [ ] plot_sgdocsvm_vs_ocsvm.py
  - [ ] plot_sgd_penalties.py
  - [ ] plot_sgd_separating_hyperplane.py
  - [ ] plot_sgd_weighted_samples.py
  - [ ] plot_sparse_logistic_regression_20newsgroups.py
  - [ ] plot_sparse_logistic_regression_mnist.py
  - [ ] plot_theilsen.py
  - [ ] plot_tweedie_regression_insurance_claims.py

- examples/manifold:
  - [ ] plot_compare_methods.py
  - [ ] plot_lle_digits.py
  - [ ] plot_manifold_sphere.py
  - [ ] plot_mds.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27599
  - [ ] plot_swissroll.py
  - [ ] plot_t_sne_perplexity.py

- examples/miscellaneous:
  - [ ] plot_anomaly_comparison.py
  - [ ] plot_display_object_visualization.py
  - [ ] plot_estimator_representation.py
  - [ ] plot_isotonic_regression.py
  - [ ] plot_johnson_lindenstrauss_bound.py
  - [ ] plot_kernel_approximation.py
  - [ ] plot_kernel_ridge_regression.py
  - [ ] plot_metadata_routing.py
  - [ ] plot_multilabel.py
  - [ ] plot_multioutput_face_completion.py
  - [ ] plot_outlier_detection_bench.py
  - [ ] plot_partial_dependence_visualization_api.py
  - [ ] plot_pipeline_display.py
  - [ ] plot_roc_curve_visualization_api.py
  - [ ] plot_set_output.py

- examples/mixture:
  - [ ] plot_concentration_prior.py
  - [ ] plot_gmm_covariances.py
  - [ ] plot_gmm_init.py
  - [ ] plot_gmm_pdf.py
  - [ ] plot_gmm.py
  - [ ] plot_gmm_selection.py
  - [ ] plot_gmm_sin.py

- examples/model_selection:
  - [ ] plot_confusion_matrix.py
  - [ ] plot_cv_indices.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27244
  - [ ] plot_cv_predict.py
  - [ ] plot_det.py
  - [ ] plot_grid_search_digits.py
  - [ ] plot_grid_search_refit_callable.py
  - [ ] plot_grid_search_stats.py
  - [ ] plot_grid_search_text_feature_extraction.py
  - [ ] plot_learning_curve.py
  - [ ] plot_likelihood_ratios.py
  - [ ] plot_multi_metric_evaluation.py
  - [ ] plot_nested_cross_validation_iris.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/28073
  - [ ] plot_permutation_tests_for_classification.py
  - [ ] plot_precision_recall.py
  - [ ] plot_randomized_search.py
  - [ ] plot_roc_crossval.py
  - [ ] plot_roc.py
  - [ ] plot_successive_halving_heatmap.py
  - [ ] plot_successive_halving_iterations.py
  - [ ] plot_train_error_vs_test_error.py
  - [ ] plot_underfitting_overfitting.py
  - [ ] plot_validation_curve.py

- examples/multioutput:
  - [ ] plot_classifier_chain_yeast.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26937

- examples/neighbors:
  - [ ] approximate_nearest_neighbors.py
  - [ ] plot_caching_nearest_neighbors.py
  - [ ] plot_classification.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26979
  - [ ] plot_digits_kde_sampling.py
  - [ ] plot_kde_1d.py
  - [ ] plot_lof_novelty_detection.py
  - [ ] plot_lof_outlier_detection.py
  - [ ] plot_nca_classification.py
  - [ ] plot_nca_dim_reduction.py
  - [ ] plot_nca_illustration.py
  - [ ] plot_nearest_centroid.py
  - [ ] plot_regression.py
  - [ ] plot_species_kde.py

- examples/neural_networks:
  - [ ] plot_mlp_alpha.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26935
  - [ ] plot_mlp_training_curves.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26935
  - [ ] plot_mnist_filters.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26935
  - [ ] plot_rbm_logistic_classification.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26935

- examples/preprocessing:
  - [ ] plot_all_scaling.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26877
  - [ ] plot_discretization_classification.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26877
  - [ ] plot_discretization.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26877
  - [ ] plot_discretization_strategies.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26877
  - [ ] plot_map_data_to_normal.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26877
  - [ ] plot_scaling_importance.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26877
  - [ ] plot_target_encoder_cross_val.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26877
  - [ ] plot_target_encoder.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26877

- examples/semi_supervised:
  - [ ] plot_label_propagation_digits_active_learning.py
  - [ ] plot_label_propagation_digits.py
  - [ ] plot_label_propagation_structure.py
  - [ ] plot_self_training_varying_threshold.py
  - [ ] plot_semi_supervised_newsgroups.py
  - [ ] plot_semi_supervised_versus_svm_iris.py

- examples/svm:
  - [ ] plot_custom_kernel.py
  - [ ] plot_iris_svc.py
  - [ ] plot_linearsvc_support_vectors.py
  - [ ] plot_oneclass.py
  - [ ] plot_rbf_parameters.py
  - [ ] plot_separating_hyperplane.py
  - [ ] plot_separating_hyperplane_unbalanced.py
  - [ ] plot_svm_anova.py
  - [ ] plot_svm_kernels.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26805
  - [ ] plot_svm_margin.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26969
  - [ ] plot_svm_nonlinear.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26972
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27108
  - [ ] plot_svm_regression.py
  - [ ] plot_svm_scale_c.py
  - [ ] plot_svm_tie_breaking.py
  - [ ] plot_weighted_samples.py

- examples/text:
  - [ ] plot_document_classification_20newsgroups.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26938
  - [ ] plot_document_clustering.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26951
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/27799
  - [ ] plot_hashing_vs_dict_vectorizer.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26967

- examples/tree:
  - [ ] plot_cost_complexity_pruning.py
  - [ ] plot_iris_dtc.py
  - [ ] plot_tree_regression_multioutput.py
  - [ ] plot_tree_regression.py
	  - [ ] https://github.com/scikit-learn/scikit-learn/pull/26962
  - [ ] plot_unveil_tree_structure.py

user: adrinjalali
labels: Meta-issue: General issue associated to an identified list of tasks
issue_26923: state: open
title: MRG Weighted v_measure_score (and related functions: entropy, contingency matrix & mutual info score)
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Fixes #26880 

#### What does this implement/fix? Explain your changes.

As pointed out in #26880, many evaluation metrics in sklearn have the ability to weight samples by some measure of importance (usually included as a `sample_weight`). This is also important to do when evaluating clustering. In particular, the `v_measure_score` (and the metrics and functions it relies on), should have this ability.

This PR adds a `sample_weight` parameter to the `homogeneity_completeness_v_measure` function (upon which the `homogeneity`, `completeness` and `v_measure_score` functions rely on, and hence they also have a `sample_weight` parameter included now). This required empowering the `entropy` calculation to take a `sample_weight`, which is straightforward with the current implementation and required minimal changes. 

It also required allowing the `contingency_matrix` to take `sample_weight`, which only means that entries are no longer integers: again, a straightforward amendment.

Finally, the `mutual_info_score`, which relies on the `contingency_matrix`, needs to allow for floating point entries in the `contingency_matrix`.

#### Any other comments?

While there appear to be quite a few changes made in this PR, it is fully backward compatible, as seen in the tests. Ample tests have also been included for those functions which now take a `sample_weight`, and they are philosophically identical to the current tests for these metrics. 

Adding this functionality will be very useful for those machine learning methods that allow for different levels of importance in contributions to the loss function from different samples (e.g. low and high energy points in a point cloud, as in the case linked in #26880 )

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: murnanedaniel
issue_26897: state: open
title: Unclear message regarding param validation
body: In the context of https://github.com/scikit-learn/scikit-learn/pull/26896 I wrote a test and got a message which I'm really puzzled about. The error message says: `ValueError: No valid specification of the columns. Only a scalar, list or slice of all integers or all strings, or boolean mask is allowed`


This is the code, and the error message:

```python
import numpy as np

from sklearn import set_config
from sklearn.model_selection import cross_validate

from sklearn.tests.test_metaestimators_metadata_routing import (
    ConsumingClassifier,
    ConsumingScorer,
    ConsumingSplitter,
)

set_config(enable_metadata_routing=True)

X = np.ones((10, 2))
y = np.array([0, 0, 1, 1, 2, 2, 3, 3, 4, 4])

scorer = ConsumingScorer().set_score_request(
    sample_weight="score_weights", metadata="score_metadata"
)
splitter = ConsumingSplitter().set_split_request(
    groups="split_groups", metadata="split_metadata"
)
estimator = ConsumingClassifier().set_fit_request(
    sample_weight="fit_sample_weight", metadata="fit_metadata"
)
n_samples = len(X)
rng = np.random.RandomState(0)
score_weights = rng.rand(n_samples)
score_metadata = rng.rand(n_samples)
split_groups = rng.randint(0, 3, n_samples)
split_metadata = rng.rand(n_samples)
fit_sample_weight = rng.rand(n_samples)
fit_metadata = rng.rand(n_samples)

cross_validate(
    estimator,
    X=X,
    y=y,
    scoring=scorer,
    cv=splitter,
    params=dict(
        score_weights=score_weights,
        score_metadata=score_metadata,
        split_groups=split_groups,
        split_metadata=split_metadata,
        fit_sample_weight=fit_sample_weight,
        fit_metadata=fit_metadata,
    ),
)
```

And the error message:

```
$ python /tmp/1.py
Traceback (most recent call last):
  File "/tmp/1.py", line 35, in <module>
    cross_validate(
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/utils/_param_validation.py", line 211, in wrapper
    return func(*args, **kwargs)
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/model_selection/_validation.py", line 406, in cross_validate
    results = parallel(
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/utils/parallel.py", line 65, in __call__
    return super().__call__(iterable_with_config)
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/joblib/parallel.py", line 1085, in __call__
    if self.dispatch_one_batch(iterator):
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/joblib/parallel.py", line 901, in dispatch_one_batch
    self._dispatch(tasks)
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/joblib/parallel.py", line 819, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 208, in apply_async
    result = ImmediateResult(func)
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 597, in __init__
    self.results = batch()
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/joblib/parallel.py", line 288, in __call__
    return [func(*args, **kwargs)
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/joblib/parallel.py", line 288, in <listcomp>
    return [func(*args, **kwargs)
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/utils/parallel.py", line 127, in __call__
    return self.function(*args, **kwargs)
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/model_selection/_validation.py", line 838, in _fit_and_score
    fit_params = _check_method_params(X, params=fit_params, indices=train)
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/utils/validation.py", line 1983, in _check_method_params
    method_params_validated[param_key] = _safe_indexing(
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/utils/__init__.py", line 341, in _safe_indexing
    indices_dtype = _determine_key_type(indices)
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/utils/__init__.py", line 288, in _determine_key_type
    raise ValueError(err_msg)
ValueError: No valid specification of the columns. Only a scalar, list or slice of all integers or all strings, or boolean mask is allowed
```

cc @jeremiedbb 
user: adrinjalali
labels: Needs Investigation: Issue requires investigation
Validation: related to input validation
issue_26892: state: open
title: Balanced Accuracy Score is NOT equal to Recall Score 
body: ### Describe the bug

By definition balanced accuracy should be equal to recall averaged over all the classes. Current implementation gives different answers. Please see the example below. 

```
import scikit.metrics as skm

y_true = [1,1]
y_pred = [1,2]

skm.recall_score(y_true, y_pred, average='macro')  # 0.25
skm.balanced_accuracy_score(y_true, y_pred)  # 0.5

```

### Steps/Code to Reproduce

```
import scikit.metrics as skm

y_true = [1,1]
y_pred = [1,2]

recall = skm.recall_score(y_true, y_pred, average='macro')  
balanced_acc = skm.balanced_accuracy_score(y_true, y_pred) 
```

### Expected Results

```
recall = balaced_acc = 0.25
```

### Actual Results

```
recall = 0.25
balanced_acc = 0.5
```

### Versions

```shell
import sklearn; sklearn.show_versions()
```

user: nisyad-ms
issue_26891: state: open
title: Putting it all together Pipelining is not working ( missing dependencies and plot )
body: ### Describe the issue linked to the documentation

From https://scikit-learn.org/stable/tutorial/statistical_inference/putting_together.html#pipelining it is missing
```
from sklearn import datasets
import numpy as np
import matplotlib.pyplot as plt
```
The last chart is missing the data

![image](https://github.com/scikit-learn/scikit-learn/assets/687311/99953518-cc6e-4a63-a1a8-b15eda3363f4)


### Suggest a potential alternative/fix

_No response_
user: diegolovison
issue_26890: state: open
title: Handling `pd.NA` in encoders
body: It seems that we don't handle it properly `pd.NA` in the encoder and thus differently than `np.nan`.
`pd.NA` will raise an error as in the following reproducible:

```python
df = pd.DataFrame({"col_1": ["A", "B", pd.NA]})
OneHotEncoder(sparse_output=False).fit_transform(df)
```

or 

```python
df = pd.DataFrame({"col_1": ["A", "B", pd.NA]})
OrdinalEncoder().fit_transform(df)
```

leading to:

<details>

```pytb
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File [~/Documents/packages/scikit-learn/sklearn/utils/_encode.py:174](https://file+.vscode-resource.vscode-cdn.net/home/glemaitre/Documents/packages/scikit-learn/~/Documents/packages/scikit-learn/sklearn/utils/_encode.py:174), in _unique_python(values, return_inverse, return_counts)
    [172](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=171) uniques_set, missing_values = _extract_missing(uniques_set)
--> [174](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=173) uniques = sorted(uniques_set)
    [175](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=174) uniques.extend(missing_values.to_list())

File [~/miniconda3/envs/dev/lib/python3.10/site-packages/pandas/_libs/missing.pyx:388](https://file+.vscode-resource.vscode-cdn.net/home/glemaitre/Documents/packages/scikit-learn/~/miniconda3/envs/dev/lib/python3.10/site-packages/pandas/_libs/missing.pyx:388), in pandas._libs.missing.NAType.__bool__()

TypeError: boolean value of NA is ambiguous

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
[/home/glemaitre/Documents/packages/scikit-learn/examples/model_selection/plot_tuned_threshold_classifier_with_metadata_routing.py](https://file+.vscode-resource.vscode-cdn.net/home/glemaitre/Documents/packages/scikit-learn/examples/model_selection/plot_tuned_threshold_classifier_with_metadata_routing.py) in line 3
      [188](file:///home/glemaitre/Documents/packages/scikit-learn/examples/model_selection/plot_tuned_threshold_classifier_with_metadata_routing.py?line=187) # %%
      [189](file:///home/glemaitre/Documents/packages/scikit-learn/examples/model_selection/plot_tuned_threshold_classifier_with_metadata_routing.py?line=188) df = pd.DataFrame({"col_1": ["A", "B", pd.NA]})
----> [190](file:///home/glemaitre/Documents/packages/scikit-learn/examples/model_selection/plot_tuned_threshold_classifier_with_metadata_routing.py?line=189) OneHotEncoder(sparse_output=False).fit_transform(df)

File [~/Documents/packages/scikit-learn/sklearn/utils/_set_output.py:140](https://file+.vscode-resource.vscode-cdn.net/home/glemaitre/Documents/packages/scikit-learn/~/Documents/packages/scikit-learn/sklearn/utils/_set_output.py:140), in _wrap_method_output..wrapped(self, X, *args, **kwargs)
    [138](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_set_output.py?line=137) @wraps(f)
    [139](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_set_output.py?line=138) def wrapped(self, X, *args, **kwargs):
--> [140](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_set_output.py?line=139)     data_to_wrap = f(self, X, *args, **kwargs)
    [141](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_set_output.py?line=140)     if isinstance(data_to_wrap, tuple):
    [142](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_set_output.py?line=141)         # only wrap the first output for cross decomposition
    [143](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_set_output.py?line=142)         return_tuple = (
    [144](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_set_output.py?line=143)             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    [145](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_set_output.py?line=144)             *data_to_wrap[1:],
    [146](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_set_output.py?line=145)         )

File [~/Documents/packages/scikit-learn/sklearn/base.py:948](https://file+.vscode-resource.vscode-cdn.net/home/glemaitre/Documents/packages/scikit-learn/~/Documents/packages/scikit-learn/sklearn/base.py:948), in TransformerMixin.fit_transform(self, X, y, **fit_params)
    [933](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=932)         warnings.warn(
    [934](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=933)             (
    [935](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=934)                 f"This object ({self.__class__.__name__}) has a `transform`"
   (...)
    [943](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=942)             UserWarning,
    [944](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=943)         )
    [946](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=945) if y is None:
    [947](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=946)     # fit method of arity 1 (unsupervised transformation)
--> [948](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=947)     return self.fit(X, **fit_params).transform(X)
    [949](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=948) else:
    [950](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=949)     # fit method of arity 2 (supervised transformation)
    [951](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=950)     return self.fit(X, y, **fit_params).transform(X)

File [~/Documents/packages/scikit-learn/sklearn/base.py:1215](https://file+.vscode-resource.vscode-cdn.net/home/glemaitre/Documents/packages/scikit-learn/~/Documents/packages/scikit-learn/sklearn/base.py:1215), in _fit_context..decorator..wrapper(estimator, *args, **kwargs)
   [1208](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=1207)     estimator._validate_params()
   [1210](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=1209) with config_context(
   [1211](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=1210)     skip_parameter_validation=(
   [1212](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=1211)         prefer_skip_nested_validation or global_skip_validation
   [1213](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=1212)     )
   [1214](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=1213) ):
-> [1215](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/base.py?line=1214)     return fit_method(estimator, *args, **kwargs)

File [~/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py:982](https://file+.vscode-resource.vscode-cdn.net/home/glemaitre/Documents/packages/scikit-learn/~/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py:982), in OneHotEncoder.fit(self, X, y)
    [972](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=971)     warnings.warn(
    [973](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=972)         (
    [974](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=973)             "`sparse` was renamed to `sparse_output` in version 1.2 and "
   (...)
    [978](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=977)         FutureWarning,
    [979](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=978)     )
    [980](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=979)     self.sparse_output = self.sparse
--> [982](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=981) self._fit(
    [983](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=982)     X,
    [984](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=983)     handle_unknown=self.handle_unknown,
    [985](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=984)     force_all_finite="allow-nan",
    [986](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=985) )
    [987](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=986) self._set_drop_idx()
    [988](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=987) self._n_features_outs = self._compute_n_features_outs()

File [~/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py:97](https://file+.vscode-resource.vscode-cdn.net/home/glemaitre/Documents/packages/scikit-learn/~/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py:97), in _BaseEncoder._fit(self, X, handle_unknown, force_all_finite, return_counts, return_and_ignore_missing_for_infrequent)
     [94](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=93) Xi = X_list[i]
     [96](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=95) if self.categories == "auto":
---> [97](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=96)     result = _unique(Xi, return_counts=compute_counts)
     [98](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=97)     if compute_counts:
     [99](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_encoders.py?line=98)         cats, counts = result

File [~/Documents/packages/scikit-learn/sklearn/utils/_encode.py:42](https://file+.vscode-resource.vscode-cdn.net/home/glemaitre/Documents/packages/scikit-learn/~/Documents/packages/scikit-learn/sklearn/utils/_encode.py:42), in _unique(values, return_inverse, return_counts)
     [11](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=10) """Helper function to find unique values with support for python objects.
     [12](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=11) 
     [13](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=12) Uses pure python method for object dtype, and numpy method for
   (...)
     [39](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=38)     array. Only provided if `return_counts` is True.
     [40](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=39) """
     [41](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=40) if values.dtype == object:
---> [42](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=41)     return _unique_python(
     [43](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=42)         values, return_inverse=return_inverse, return_counts=return_counts
     [44](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=43)     )
     [45](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=44) # numerical
     [46](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=45) return _unique_np(
     [47](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=46)     values, return_inverse=return_inverse, return_counts=return_counts
     [48](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=47) )

File [~/Documents/packages/scikit-learn/sklearn/utils/_encode.py:179](https://file+.vscode-resource.vscode-cdn.net/home/glemaitre/Documents/packages/scikit-learn/~/Documents/packages/scikit-learn/sklearn/utils/_encode.py:179), in _unique_python(values, return_inverse, return_counts)
    [177](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=176) except TypeError:
    [178](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=177)     types = sorted(t.__qualname__ for t in set(type(v) for v in values))
--> [179](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=178)     raise TypeError(
    [180](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=179)         "Encoders require their input to be uniformly "
    [181](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=180)         f"strings or numbers. Got {types}"
    [182](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=181)     )
    [183](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=182) ret = (uniques,)
    [185](file:///home/glemaitre/Documents/packages/scikit-learn/sklearn/utils/_encode.py?line=184) if return_inverse:

TypeError: Encoders require their input to be uniformly strings or numbers. Got ['NAType', 'str']
```

</details>

If `np.nan` is used, then we are fine:

```python
df = pd.DataFrame({"col_1": ["A", "B", np.nan]})
OneHotEncoder(sparse_output=False).fit_transform(df)
```

```
array([[1., 0., 0.],
       [0., 1., 0.],
       [0., 0., 1.]])
```

```python
df = pd.DataFrame({"col_1": ["A", "B", np.nan]})
OrdinalEncoder().fit_transform(df)
```

```
array([[ 0.],
       [ 1.],
       [nan]])
```
user: glemaitre
issue_26888: state: open
title: ENH Add `np.float32` data support for `HDBSCAN`
body: #### Reference Issues/PRs
Towards https://github.com/scikit-learn/scikit-learn/issues/26801

#### What does this implement/fix? Explain your changes.
Adds support for calculations using `np.float32` to avoid unnecessary copies and memory bloat when lower precision is tolerable.

#### Any other comments?
Currently waiting on the completion of https://github.com/scikit-learn/scikit-learn/pull/25914 and subsequent work
user: Micky774
issue_26887: state: open
title: infinity in split nodes
body: ### Describe the bug

![image](https://github.com/scikit-learn/scikit-learn/assets/72708109/05146230-843e-4347-92cb-aed5e642328a)

I've created a simple model with decision trees. However, when I used the "export_text" function, I noticed that there are "infinity" values in the split nodes. Is this an error, or is it an expected behavior? It doesn't seem logical to me at all.

note: train data contains null.

### Steps/Code to Reproduce

DT = sklearn.tree.DecisionTreeClassifier()
DT.fit(X_train, y_train)
export_text(DT)

### Expected Results

There shouldn't infinity.

### Actual Results

in image.

### Versions

```shell
System:
    python: 3.11.4 (tags/v3.11.4:d2340ef, Jun  7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)]
executable: C:\Users\sametc\AppData\Local\Programs\Python\Python311\python.exe
   machine: Windows-10-10.0.19044-SP0

Python dependencies:
      sklearn: 1.3.0
          pip: 23.1.2
   setuptools: 68.0.0
        numpy: 1.25.1
        scipy: 1.10.1
       Cython: 3.0.0b3
       pandas: 2.0.2
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: C:\Users\sametc\AppData\Local\Programs\Python\Python311\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\sametc\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.23-gcc_10_3_0.dll
        version: 0.3.23
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\sametc\AppData\Local\Programs\Python\Python311\Lib\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8
```

user: sametcopur
labels: Needs Reproducible Code: Issue requires reproducible code
Needs Investigation: Issue requires investigation
issue_26880: state: open
title: Add sample weighting to V Measure Score
body: ### Describe the workflow you want to enable

Many evaluation metrics in sklearn have the ability to weight samples by some measure of importance (usually included as a `sample_weight`). This is also important to do when evaluating clustering. In particular, the `v_measure_score` (and the metrics and functions it relies on), should have this ability.

### Describe your proposed solution

The proposed solution is to amend the `v_measure_score` function, as well as the `entropy`, the `contingency_matrix` and the `mutual_info_score` functions, to include a `sample_weight` input parameter. 

This has been implemented and tested already in [this](https://github.com/wpmccormack/ePIC_Clustering_2023/blob/pip_installable/epic_clustering/scoring/scoring_functions.py) repository, for [this](https://www.kaggle.com/competitions/hgs-hire-power-week-an-epic-competition/overview) Kaggle challenge scoring metric. 

Making the changes requires very little substantive changes to each function, and handling the edge cases is straightforward. This would make the `v_measure_score` much more useful for cases such as those in the Kaggle challenge (clustering of energy deposits of different intensities).

### Describe alternatives you've considered, if relevant

A possible alternative is to keep the current functions, and handle weighting by repeating entries according to their weight. However, this is expensive and clunky, and doesn't work for fractional weights.

### Additional context

_No response_
user: murnanedaniel
issue_26860: state: open
title: MAINT create y before enforcing X using tags
body: We should create `y` before enforcing rules on `X`. This could be problematic for third-party libraries. I got issues in `skrub` where `X` was enforced to have `string` only and create `y` after the enforcement does not work.
user: glemaitre
issue_26848: state: open
title: Ridge and RidgeCV has different result even if alpha is the same
body: ### Describe the bug

(From discussion in https://github.com/scikit-learn/scikit-learn/discussions/26733)

In one of my dataset, I try to specify only one value in alphas in RidgeCV, and expects RidgeCV should give me the same result as using Ridge with same alpha value. But it would not (see the code and output below). And the RidgeCV result seems abnormal.

I found that the problem is related to some large values we used in sample_weight. I can reproduce it with make_regression.

I guess the large value in sample_weight may caused some overflow in RidgeCV intermediate calculation?
Given that RidgeCV and Ridge are using different solvers under the hood, I think it is not surprise that they may behave differently in this case? I just scaled down the weight and the problem is gone.



### Steps/Code to Reproduce

```python 
from sklearn.datasets import make_regression

X, y = make_regression(n_features=2)
wgt = np.random.randint(2e5, 2e11, size=100)

print('=== RidgeCV ===')
clf = RidgeCV(alphas=[0.001],fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt,
)
display(f'{clf.coef_ = }')
display(f'{clf.intercept_ = }')
display(f'{clf.alpha_ = }')

print('=== Ridge ===')
clf = Ridge(alpha=0.001,fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt
)
display(f'{clf.coef_ = }')
display(f'{clf.intercept_ = }')
```

### Expected Results

The output of RidgeCV and Ridge is the same.

### Actual Results

```
=== RidgeCV ===
'clf.coef_ = array([-95.63781897,  31.72140137])'
'clf.intercept_ = 2.6375015138569937'
'clf.alpha_ = 0.001'
=== Ridge ===
'clf.coef_ = array([41.8887401, 33.131992 ])'
'clf.intercept_ = -4.440892098500626e-16'
```

### Versions

```shell
I am using scikit-learn 1.2.1 on python 3.8 on Linux.
```

user: johnlcf
labels: Needs Investigation: Issue requires investigation
issue_26842: state: open
title: `check_estimator` fails when checked after doing `set_output(transform='pandas')` on the instance
body: ### Describe the bug

`sklearn.utils.estimator_checks.check_estimator` fails with a method when `set_output` is called(`<estimator_instance>.set_output(transform='pandas')`), but doesn't fail when it is not called.

I was doing the test on a custom estimator that I had written and after fail, I wanted to check the behaviour on an estimator that is part of sklearn and that's when I get this error. So, I will adopt the fix that I get from this issue.

### Steps/Code to Reproduce
```
import pandas as pd
from sklearn import preprocessing
from sklearn.datasets import fetch_california_housing
from sklearn.utils.estimator_checks import check_estimator

cal_house = fetch_california_housing(as_frame=True)
cal_house = pd.merge(left=cal_house['data'], right=cal_house['target'], left_index=True, right_index=True)

my_stand_scale = preprocessing.StandardScaler()
my_stand_scale.set_output(transform='pandas')
my_stand_scale.fit_transform(X=cal_house.drop(columns='MedHouseVal', axis=1), y=cal_house.loc[:,'MedHouseVal'])

check_estimator(estimator=my_stand_scale, generate_only=False)
```

### Expected Results

No error is thrown by `check_estimator`

### Actual Results
```
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
[<ipython-input-12-1f43d30ea78e>](https://localhost:8080/#) in <cell line: 12>()
     10 my_stand_scale.fit_transform(X=cal_house.drop(columns='MedHouseVal', axis=1), y=cal_house.loc[:,'MedHouseVal'])
     11 
---> 12 check_estimator(estimator=my_stand_scale, generate_only=False)

2 frames
[/usr/local/lib/python3.10/dist-packages/sklearn/utils/estimator_checks.py](https://localhost:8080/#) in check_estimator(estimator, generate_only, Estimator)
    631     for estimator, check in checks_generator():
    632         try:
--> 633             check(estimator)
    634         except SkipTest as exception:
    635             # SkipTest is thrown when pandas can't be imported, or by checks

[/usr/local/lib/python3.10/dist-packages/sklearn/utils/_testing.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    186             with warnings.catch_warnings():
    187                 warnings.simplefilter("ignore", self.category)
--> 188                 return fn(*args, **kwargs)
    189 
    190         return wrapper

[/usr/local/lib/python3.10/dist-packages/sklearn/utils/estimator_checks.py](https://localhost:8080/#) in check_no_attributes_set_in_init(name, estimator_orig)
   3094     # Test for no setting apart from parameters during init
   3095     invalid_attr = set(vars(estimator)) - set(init_params) - set(parents_init_params)
-> 3096     assert not invalid_attr, (
   3097         "Estimator %s should not set any attribute apart"
   3098         " from parameters during init. Found attributes %s."

AssertionError: Estimator StandardScaler should not set any attribute apart from parameters during init. Found attributes ['_sklearn_output_config'].
```
### Versions

```shell
scikit-learn==1.2.2 on Python 3.10.12 on Ubuntu 20.04.6 LTS (or google colab on date this issue was created.)
```

user: naveen-marthala
issue_26839: state: open
title: ENH add pos_label to confusion_matrix
body: While working on https://github.com/scikit-learn/scikit-learn/pull/26120, we figure out that we don't have a `pos_label` parameter in the `confusion_matrix` function. To follow the documentation and since we don't report the column/index name, we should not change the positions of the TP/TN/FP/FN.

`pos_label` allows flipping the matrix in case this is not the expected positive label.

I also added the `pos_label` to the `ConfusionMatrixDisplay` to be complete.
user: glemaitre
issue_26837: state: open
title: FIX do not report wrong iteration number when initialization of EM does not converge
body: Fixes #26621

When the EM best iteration does not converge, a warning message is raised, indicating a wrong iteration number. As discussed in #26621, we don't need to indicate the number but rather mention that the convergence did not happen.
user: krstopro
issue_26835: state: open
title: Transform output to xarray objects
body: ### Describe the workflow you want to enable

I just watched @thomasjpfan's SciPy lightning talk where he showed using `.set_output(transform='pandas')` to return results wrapped in pandas dataframes. **I want the same but for `xarray.Dataset`**.

### Describe your proposed solution

However it works for pandas but for xarray. (Which PR was it implemented in?) There is even an [`xarray.Dataset.from_dataframe`](https://docs.xarray.dev/en/latest/generated/xarray.Dataset.from_dataframe.html) method.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I think a fully N-dimensional object would make a lot of sense to wrap sklearn output in. I recently found myself wishing this existed whilst doing PCA for example.
user: TomNicholas
issue_26821: state: open
title: Add `strata` to `Stratified*Split` CV splitters
body: Right now `Stratified*Split` classes take `y` as the strata, while that's not always the case.

In `train_test_split` we allow a `stratify` arg (which I'm wondering if it should be called `strata`), which defines the groups samples belong to. And inside, we basically do this:

```py
    cv = StratifiedShuffleSplit(test_size=n_test, train_size=n_train, random_state=random_state)

    train, test = next(cv.split(X=arrays[0], y=stratify))

    return list(
        chain.from_iterable(
            (_safe_indexing(a, train), _safe_indexing(a, test)) for a in arrays
        )
    )
```

As you can see, we're passing `stratify` as `y` to the splitter. I think it would make sense to add a `strata` arg to the splitters, and if `None`, we'd take values in `y` instead, as it is now.

Note that now that we have SLEP6, we won't need a separate class for them, and we'd simply request `strata` for the splitter:

```py
cv = StratifiedShuffleSplit().set_split_request(strata=True)
...
GridSearchCV(model, param_grid, cv=cv).fit(X, y, strata=strata_values)
cross_validate(model, X, y, cv=cv, props={"strata": strata_values})
```

cc @marenwestermann 
user: adrinjalali
issue_26820: state: open
title: PERF speed up confusion matrix calculation
body: Fixes https://github.com/scikit-learn/scikit-learn/issues/26808

Speed up `confusion_matrix` calculation.

This achieves the speedup by calculating the unique values on `y_true` and `y_pred` only once, and adds the required args along the way to the inner methods.

Not sure about the names of the added args on public functions, and might need to refactor to hide them if necessary.

Also this needs to add the same for some other functions to make the tests pass, but for now this works, and brings the time from about 15s to 5s on my machine.

```py
import numpy as np

from sklearn.metrics import classification_report

y_true = np.random.randint(0, 2, size=2**23)
y_pred = y_true.copy()
np.random.shuffle(y_pred[2**20 : 2**21])

print(
    classification_report(
        y_true=y_true,
        y_pred=y_pred,
        digits=10,
        output_dict=False,
        zero_division=0,
    )
)
```
user: adrinjalali
issue_26809: state: open
title: DOC Update web site to pydata-sphinx theme.
body: #### What does this implement/fix? Explain your changes.

Dear maintainers,
this draft pull request updates the theme of the website to the [pydata-sphinx](https://github.com/pydata/pydata-sphinx-theme/) theme.
This was suggested [here](https://github.com/scikit-learn/scikit-learn/issues/16951#issuecomment-1604350652). 

Thanks for considering it.

#### Any other comments?

The pull request is a test to check how smooth would be the style conversion, a lot of details need to be fixed but in my opinion the result is already really nice... kudos to the pydata-sphinx theme developers!

I had to update dependencies to add the sphinx theme: my understanding was that I had to run `update_environments_and_lock_files.py` to do that in an automatic way. This ended up in updating the lock files too and I'm not sure it is needed. I was looking for a way to have the CI working ... Is this the way? 

Perhaps @lucyleeow would be interested in having a look and move forward together? :innocent: 

### Functionalities checklist
- [ ] home page carousel
- [ ] what's new labels
- [x] dark theme (see [comment](https://github.com/scikit-learn/scikit-learn/pull/26809#issuecomment-1628571546))
- [x] top bar links (see [comment](https://github.com/scikit-learn/scikit-learn/pull/26809#issuecomment-1628571546))
~~- [ ] old version switch (see [comment](https://github.com/scikit-learn/scikit-learn/pull/26809#issuecomment-1628635403))~~
user: cmarmo
issue_26808: state: open
title: Speed up classification_report
body: ### Describe the workflow you want to enable

I'm concerned with slow execution speed of the classification_report  procedure which makes it barely suitable for production-grade workloads.
On a 8M sample it already takes 15 seconds whereas simple POC numba implementation takes 20 to 40 **milli**seconds. I understand that sklearn code is well covered with tests, has wide functionality, follows style guildlines and best practices, but there should be no excuse for the performace leap with simple POC of magnitude that big (x1000).

```python
import numpy as np
from sklearn.metrics import classification_report

y_true = np.random.randint(0, 2, size=2 ** 23)
y_pred = y_true.copy()
np.random.shuffle(y_pred[2 ** 20 : 2 ** 21])

print(classification_report(y_true=y_true, y_pred=y_pred, digits=10, output_dict=False, zero_division=0,))
```

>               precision    recall  f1-score   support
> 
>            0  0.9373906697 0.9373906697 0.9373906697   4192570
>            1  0.9374424159 0.9374424159 0.9374424159   4196038
> 
>     accuracy                      0.9374165535   8388608
>     macro avg  0.9374165428 0.9374165428 0.9374165428   8388608
>     weighted avg  0.9374165535 0.9374165535 0.9374165535   8388608
> 
> time: 18.6 s (started: 2023-07-08 16:10:08 +03:00)
> 


`print(own_classification_report(y_true=y_true, y_pred=y_pred, zero_division=0))`


> (3930076, 3933544, 262494, 262494, 0.9374165534973145, array([4192570, 4196038], dtype=int64), array([0.93739067, 0.93744242]), array([0.93739067, 0.93744242]), array([0.93739067, 0.93744242]))
> time: 16 ms (started: 2023-07-08 16:11:18 +03:00)

### Describe your proposed solution

```python
import numpy as np
from numba import njit


@njit()
def own_classification_report(y_true: np.ndarray, y_pred: np.ndarray, nclasses: int = 2, zero_division: int = 0):

    correct = np.zeros(nclasses, dtype=np.int64)
    wrong = np.zeros(nclasses, dtype=np.int64)
    for truth, pred in zip(y_true, y_pred):
        if pred == truth:
            correct[pred] += 1
        else:
            wrong[pred] += 1
    tn, tp = correct
    fn, fp = wrong
    accuracy = (tn + tp) / len(y_true)

    groups = np.array([(tn + fn), (tp + fp)])
    supports = np.array([(tn + fp), (tp + fn)])

    precisions = np.array([tn, tp]) / groups
    recalls = np.array([tn, tp]) / supports
    f1s = 2 * (precisions * recalls) / (precisions + recalls)

    for arr in (precisions, recalls, f1s):
        np.nan_to_num(arr, copy=False, nan=zero_division)

    return tn, tp, fn, fp, accuracy, supports, precisions, recalls, f1s
```

### Describe alternatives you've considered, if relevant

Can someone from original classification_report authors take a look please why it's taking so long?

### Additional context

I have the latest versions of scikit-learn and numba to this date, an Intel CPU with AVX.
user: fingoldo
labels: Needs Investigation: Issue requires investigation
issue_26802: state: open
title: ⚠️ CI failed on Linux_Runs.pylatest_conda_forge_mkl ⚠️
body: **CI is still failing on [Linux_Runs.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=64500&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a)** (Feb 27, 2024)
- Test Collection Failure
user: scikit-learn-bot
issue_26801: state: open
title: HDBSCAN Ongoing Work
body: ## Introduction

This is a (hopefully) exhaustive list of ongoing/future work for HDBSCAN. These have all been discussed and are considered wanted, but some still require thorough investigation (especially heuristic evaluations).

## Priority List

The higher priority items appear earlier in this list.

- [ ] https://github.com/scikit-learn/scikit-learn/pull/26888
    - [x] https://github.com/scikit-learn/scikit-learn/pull/25914
    - [ ] Finalize `{KD, Ball}Tree` API to avoid writing custom dispatcher
- [ ] Support `np.nan` in Cython implementation for sparse matrices
    - [x] https://github.com/scikit-learn/scikit-learn/pull/26889
- [ ] [Reintroduce `Boruvka` algorithm](https://github.com/scikit-learn/scikit-learn/pull/27572) (removed in https://github.com/scikit-learn/scikit-learn/commit/b7736ef6db1650ba4c3d8d830348aacfe5589015)
- [ ] Implement PWD backend for weighted `argkmin` in medoid calculation
- [ ] Investigate PWD backend for `mst_from_*` functions in `_linkage.pyx`
- [ ] Investigate PWD backend for `_reachability.pyx`
- [ ] Benchmark KD vs Ball Tree efficiency
- [ ] Add consistent threading semantics to enable `prange`, e.g. in `_reachability.pyx`
- [ ] Improve partition strategy in `_reachability.pyx` (cf. https://github.com/scikit-learn/scikit-learn/pull/24701#discussion_r1035775220)
- [ ] Add support for `np.inf` values when `metric=='precomputed'` and `X` is sparse.

user: Micky774
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_26794: state: open
title: Handle np.nan / missing values in KBinsDiscretizer
body: Especially when the output is one-hot encoded, it would be quite natural to encode missing values as zeros or add an extra missingness indicator feature.

Related to:

- https://github.com/scikit-learn/scikit-learn/issues/26793

user: ogrisel
issue_26793: state: open
title: Handle np.nan / missing values in SplineTransformer
body: I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.
user: ogrisel
issue_26775: state: open
title: making mutual_info_regression more verbose
body: ### Describe the workflow you want to enable

Hi, 

Thanks for coding up `mutual_info_regression`, I use it for novel space science research and it's super helpful for me.

I'm currently using it to analyse quite large amounts of data, and understandably this can be quite slow. I'm wondering if it would be possible to implement an optional flag in `mutual_info_regression` to make the code verbose while it's running. For example, printing out something to the terminal when it is 10%, 20%, 30% (etc) done.

Just an idea - hopefully I have raised this in the correct way!

Thanks,
Alexandra Fogg

### Describe your proposed solution

Implementation of an optional flag in `mutual_info_regression` to make the function verbose, where it prints out statements at certain stages of the process, e.g. 10% done or saying that a certain stage is completed, etc.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: arfogg
issue_26758: state: open
title: Build in an option to change the "positive" class in sklearn.metrics.precision_recall_curve and sklearn.metrics.roc_curve
body: ### Describe the workflow you want to enable

In some cases, it is nice to compare a machine learning classifier with experimental data using ROC or Precision-Recall Curves. For (e.g.) a logarithmic regression model, the score returned by the model will be a probability value or a value from the corresponding decision function. Both values are positively correlated with the "positive" class and can be readily used for calculation of fpr and tpr using sklearn.metrics.precision_recall_curve or sklearn.metrics.roc_curve. However, experimental data can also be negatively correlated with the "positive class". Thus, when calculating confusion matrices, one would need to assume that the "positive" events are not at the right side of the threshold, but on the left side. **The same problem occurs for imbalanced test data sets.** Here, it is useful to use precision-recall curves and to define the "positive" class as the "minority class" (see https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-imbalanced-classification/). However, in some cases, the minority class is negatively correlated with the probability values or decision function values of the classifier (i.e., the minority class corresponds to a probability of zero or to negative decision funtion values). Than it is not enough to just use the "pos_label" option as again "positive" values are on the left and not on the right side of the thresholds used for calculating the confusion matrices/precision-recall curve (see above).

### Describe your proposed solution

One could add a parameter that is given to sklearn.metrics.precision_recall_curve and sklearn.metrics.roc_curve. This parameter descibes whether the `y_score` paramater is positively or negatively correlated with the "positive" class (something like `pos_corr=String, default='psoitive'` ; the options would be 'positive' or 'negative'). The functions would than get an additional if condition:

```
y_score=np.array(y_score)
if pos_corr == 'negative':
    y_score=y_score*(-1)

```
This would transform a negative correlation to a positive one and sklearn.metrics.precision_recall_curve and sklearn.metrics.roc_curve could be used without further changes.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: ThorbenMaa
labels: Needs Investigation: Issue requires investigation
issue_26735: state: open
title: GaussianMixture with BIC/AIC
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #19338. Automates the selection in [Gaussian Mixture Model Selection](https://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_selection.html). Adds a basic `GaussianMixtureIC` estimator without initializing with agglomerative clustering as discussed in #19562.

#### What does this implement/fix? Explain your changes.
It automatically selects the best GM model based on BIC or AIC among a set of models that are parameterized by:

- Covariance constraints

- Number of components

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: tingshanL
issue_26729: state: open
title: FIX do not allow for p=None in NN-based algorithm
body: From the discussion in https://github.com/scikit-learn/scikit-learn/pull/26568/files#r1246511365, it seems that we should not accept `p=None` because it raises an error:

```pytb
TypeError: '<' not supported between instances of 'NoneType' and 'int'
```

with the minimal reproducer:

```python
import numpy as np
from sklearn.neighbors import NearestNeighbors
samples = [[0, 0, 2], [1, 0, 0], [0, 0, 1]]
neigh = NearestNeighbors(n_neighbors=2, radius=0.4, p=None)
neigh.fit(samples)
```

Let's see if we break something somewhere else depending on some combination of parameters.
user: glemaitre
issue_26724: state: open
title: Array API support with lazy evaluation
body: At the moment, our Array API integration can implicitly assume eager evaluation semantics.

Furthermore we did not test our code to see how it would behave with candidate Array API implementations with lazy evaluation semantics (e.g. dask, jax, ...).

The purpose of this issue is to track what would be needed to make our Array API estimator work with lazy evaluation.

I think we should first investigate where this breaks, then decide whether or not we would like to support lazy evaluation semantics in scikit-learn via the Array API support and if so open a meta-issue to add common test with an estimator tag and progressively fix the Array API estimators to deal with lazy evaluation.

Note that are particular point about lazy vs eager evaluation for Array API consuming libraries is being discussed here:

- https://github.com/data-apis/array-api/issues/642
user: ogrisel
issue_26711: state: open
title: AttributeError: This 'LabelEncoder' has no attribute 'set_output'
body: ### Describe the bug

I tried to call **'set_output'** from LabelEncoder object and got the AttributeError.

[The document](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html) says sklearn.preprocessing.LabelEncoder has **'set_output'** method, but it was not working.

Soon I found most of other **'set_output'** available estimators inherits both of sklearn.base.OneToOneFeatureMixin and sklearn.base.TransformerMinxin

Howerver, [LabelEncoder](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/preprocessing/_label.py) only inherits the TransformerMinxin.

```python
class LabelEncoder(TransformerMixin, BaseEstimator):
```

</br>

Function **'set_output'** seems available when **'_auto_wrap_is_configured'** is True. [(utils._set_output.py)](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/_set_output.py)

```python
    @available_if(_auto_wrap_is_configured)
    def set_output(self, *, transform=None):
        """Set output container.

        See :ref:`sphx_glr_auto_examples_miscellaneous_plot_set_output.py`
        for an example on how to use the API.

        Parameters
        ----------
        transform : {"default", "pandas"}, default=None
            Configure output of `transform` and `fit_transform`.

            - `"default"`: Default output format of a transformer
            - `"pandas"`: DataFrame output
            - `None`: Transform configuration is unchanged

        Returns
        -------
        self : estimator instance
            Estimator instance.
        """
        if transform is None:
            return self

        if not hasattr(self, "_sklearn_output_config"):
            self._sklearn_output_config = {}

        self._sklearn_output_config["transform"] = transform
        return self
```

</br>

Then estimator should have **'get_feature_names_out'** to make **'_auto_wrap_is_configured'** returns True. [(utils._set_output.py)](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/_set_output.py)

```python
def _auto_wrap_is_configured(estimator):
    """Return True if estimator is configured for auto-wrapping the transform method.

    `_SetOutputMixin` sets `_sklearn_auto_wrap_output_keys` to `set()` if auto wrapping
    is manually disabled.
    """
    auto_wrap_output_keys = getattr(estimator, "_sklearn_auto_wrap_output_keys", set())
    return (
        hasattr(estimator, "get_feature_names_out")
        and "transform" in auto_wrap_output_keys
    )
```

</br>

 To have **'get_feature_names_out'** attr, estimator should inherit OneToOneFeatureMixin as I think. [(base.py)](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/base.py)

```python
class OneToOneFeatureMixin:
    """Provides `get_feature_names_out` for simple transformers.

    This mixin assumes there's a 1-to-1 correspondence between input features
    and output features, such as :class:`~preprocessing.StandardScaler`.
    """

    def get_feature_names_out(self, input_features=None):
        """Get output feature names for transformation.

        Parameters
        ----------
        input_features : array-like of str or None, default=None
            Input features.

            - If `input_features` is `None`, then `feature_names_in_` is
              used as feature names in. If `feature_names_in_` is not defined,
              then the following input feature names are generated:
              `["x0", "x1", ..., "x(n_features_in_ - 1)"]`.
            - If `input_features` is an array-like, then `input_features` must
              match `feature_names_in_` if `feature_names_in_` is defined.

        Returns
        -------
        feature_names_out : ndarray of str objects
            Same as input features.
        """
        check_is_fitted(self, "n_features_in_")
        return _check_feature_names_in(self, input_features)
```

</br>

I want to know that it is kind of a bug, or the document says wrong information.

### Steps/Code to Reproduce

```python
from sklearn.preprocessing import LabelEncoder

LabelEncoder().set_output()
```

### Expected Results

No error is thrown

### Actual Results

```python
--------------------------------------------------------------------------------
AttributeError                                 Traceback (most recent call last)                        
/tmp/ipykernel_16765/3091610596.py in <module>
---> 1 LabelEncoder().set_output()

~/.local/lib/python3.8/site-packages/sklearn/utils/_available_if.py in __get__(self, obj, owner)
      31        # this is to allow access to the docstrings.
      32        if not self.check(obj):
--->  33            raise attr_err
      34        out = MethodType(self.fn, obj)
      35
AttributeError: This 'LabelEncoder' has no attribute 'set_output'
```
            
             
            

### Versions

```shell
1.2.2
```

user: Clarit7
issue_26707: state: open
title: BENCH newton-lsmr solver for GLMs - inner stopping criterion
body: #### Reference Issues/PRs
This PR is **not for merge**, but for analysis of #25462.

The newton-lsmr solver needs a good inner stopping criterion with a good forcing sequence. This is investigated here and benchmarks are run.

For stopping criteria of inexact Newton methods (mainly conjugate gradient for inner solver) see:
- Stephen G. Nash "A survey of truncated-Newton methods" https://doi.org/10.1016/s0377-0427(00)00426-x
- Galli & Lin "A Study on Truncated Newton Methods for Linear Classification"
  https://www.csie.ntu.edu.tw/~cjlin/papers/tncg/tncg.pdf
  https://doi.org/10.1109/TNNLS.2020.3045836

#### Any other comments?
See added notebook [Bench_Newton_LSMR.ipynb](https://github.com/scikit-learn/scikit-learn/blob/56e495344000b8b8d4cff8237a4c702489919b39/Bench_Newton_LSMR.ipynb) in this PR for analysis results.
user: lorentzenchr
issue_26701: state: open
title: DOC improve section folding behaviour
body: Bring some improvements to the new foldable sections:

- [x] Use larger padding as in Bootstrap
- [x] Use a small animation to expand details
- [x] Add an additional option to expand all details at once

I did not want to change the `<details>`/`<summary>` pattern since the find-in-page already works with Chrome and Edge. We can expect this feature to land on Firefox at some point.
user: glemaitre
issue_26692: state: open
title: [MRG] MAINT Allow estimators to opt-in to complex numbers in `check_array`
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Fixes #25922.


#### What does this implement/fix? Explain your changes.

This allows scikit-learn estimators to opt-in to allowing complex numbers, via a new flag to `check_array`: `accept_complex`.

By default, `accept_complex=False`, meaning all validations **remain unchanged by default**. This new parameter simply allows estimators to turn off this check.

#### Any other comments?

The current BaseEstimator does not allow downstream estimators to use complex numbers, or give them any option to turn this off.

For example, in [PySR](https://github.com/MilesCranmer/PySR)'s PySRRegressor (which inherits `BaseEstimator`), we need to allow complex numbers, and thus had to automatically monkey patch this into scikit-learn on import: https://github.com/MilesCranmer/PySR/blob/master/pysr/sklearn_monkeypatch.py. This is obviously not a great solution, but it's our only option until there is a flag to turn off the complex number check.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


- [x] Option to disable on `check_array`.
- [x] Option to disable on `check_X_y`.
- [x] Add unit tests.
- [x] Update changelog.
user: MilesCranmer
issue_26691: state: open
title: ENH run common tests for SparseCoder
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Fixes [https://github.com/scikit-learn/scikit-learn/issues/26482](https://github.com/scikit-learn/scikit-learn/issues/26482)

#### What does this implement/fix? Explain your changes.

This PR runs `SparseCoder` common tests by accepting `dictionary` as`required_parameters` in `utils.estimator_cheks._construct_instance`.

* Constructs `SparseCoder` instance in `utils.estimator_cheks._construct_instance`  function
* Adds parameter validation for `SparseCoder`
* Adds stateless tag for `SparseCoder`
* Fixes data type preservation for` SparseCoder.transform()` which was failing during common tests
* Test modifications were needed as as `n_features_in` of input `X` should match `n_features` of  `precomputed_dictionary `of `SparseCoder`
* Some `get_features_out `tests were skipped for SparseCoder due to SparseCoder's `feature_names_in `attribute not resetting based on input.

#### Any other comments?



<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Shreesha3112
issue_26689: state: open
title: ENH Add decision_function, predict_proba and predict_log_proba for NearestCentroid estimator
body: #### Reference Issues/PRs
closes #17711 and #26659
supersede #17711 

#### What does this implement/fix? Explain your changes.
Add `decision_function`, `predict_proba` and `predict_log_proba` methods for **NearestCentroid** estimator class for usage with `roc_curve` function.

The **NearestCentroid** class constructor now has an additional parameter called _priors_. By default, _priors_ is set to `None`. When _priors_ is set to `None`, class _priors_ will be estimated using the sample data _X_. Thus, it is not backward compatible with the old version of **NearestCentroid**. The old and new **NearestCentroid** estimators will yield different results because the old **NearestCentroid** estimator assumed equal class _priors_.

#### Any other comments?
I try to keep the design as consistent as possible with the **LinearDiscriminantAnalysis** estimator. So, the _priors_ parameter in both classes are the same. Open to feedback and advise.

#### References

Tibshirani, R., Hastie, T., Narasimhan, B., & Chu, G. (2002). Class Prediction by Nearest Shrunken Centroids, with Applications to DNA Microaarrays. Statistical Science 18(1), p. 104-117.


user: NoPenguinsLand
issue_26687: state: open
title: Small doc change to help people finding (via browser search) content in a fold
body: Based on a comment by @adrinjalali : improve things slightly.
user: GaelVaroquaux
issue_26686: state: open
title: MAINT Move stuff outside of utils.__init__
body: Fixes https://github.com/scikit-learn/scikit-learn/issues/11633

`utils.__init__` contains a bunch of unrelated functions, some public, some private.
This PR proposes to move them into dedicated submodules (creating `_indexing.py`, `_missing.py`, `_optional_dependencies.py` and `_misc.py`).

Part of the motivation is to help developing `utils` without having to work around circular imports, like what's happening in https://github.com/scikit-learn/scikit-learn/pull/26648#discussion_r1237104026 

there are a few functions that I did not know where to put so I put them in a `_misc.py` submodule. I still think it's better than having the whole `utils.__init__` act as a big `_misc` module.
user: jeremiedbb
issue_26684: state: open
title: ENH `warnings.warn` stacklevel
body: Currently many warnings in scikit-learn use `stacklevel=1`, which in most cases will only print out `warnings.warn(`. I think we should set `stacklevel=2` to at least get something more informative than `stacklevel=1`?

I've also noticed that pandas implements a `find_stack_level()` function which finds the first place in the stack that is not inside pandas. Maybe we can have a similar implementation in scikit-learn, but I'm also aware that this may cause preformance regression especially since warnings may be raised in loops in scikit-learn.

I can make a PR for either suggestion (if desired). What do maintainers think?
user: Charlie-XIAO
issue_26659: state: open
title: Implement probability estimates and decision function for NearestCentroid classifier
body: ### Describe the workflow you want to enable

Related to still open #17711, add implementation of .predict_proba() and/or .decision_function().

### Describe your proposed solution

Distance function should follows equation 18.2 (with shrunken or unshrunken centroids) and class probability estimates should follows equation 18.8, both from the 2nd edition of Elements of Statistical Learning (print 12).

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I'm trying to compute/plot ROC curve for the NearestCentroid classifier and got the following error:
```ValueError: response method decision_function or predict_proba is not defined in NearestCentroid```
user: NoPenguinsLand
issue_26658: state: open
title: Automatic bandwidth calculation valid only for normalized data
body: ### Describe the bug

`sklearn.neighbors.KernelDensity` supports automatic (optimal) bandwidth calculation via `bandwidth = 'silverman'` and `bandwidth = 'scott'`. The algorithm computes the appropriate observation-weighted bandwidth factors (proportional to nobs^0.2) but does not adjust for the standard deviation or interquartile range of the dataset. Roughly, the algorithm should scale the dataset's standard error by the algorithmic bandwidth factors.

See, e.g., [Wikipedia](https://en.wikipedia.org/wiki/Kernel_density_estimation#A_rule-of-thumb_bandwidth_estimator). The implementation in `scipy.stats._kde` is correct.

### Steps/Code to Reproduce
```python
import matplotlib.pyplot as plot
import numpy as np
from sklearn.neighbors import KernelDensity
from scipy.stats import gaussian_kde

data = np.random.normal( scale = 0.01, size = 100 )

#
# 1. sklearn (auto)
#
kd_sklearn_auto = KernelDensity( kernel = 'gaussian', bandwidth = 'silverman' )
kd_sklearn_auto.fit( np.reshape( data, ( -1, 1 ) ) )

#
# 2. sklearn (manual)
#
kd_sklearn_manual = KernelDensity( kernel = 'gaussian', bandwidth = 0.9 * np.std( data ) / len( data ) ** ( 1 / 5 ) )
kd_sklearn_manual.fit( np.reshape( data, ( -1, 1 ) ) )

#
# 3. scipy
#
kd_scipy = gaussian_kde( data, bw_method = 'silverman' )

#
# 4. show the difference
#
xs = np.arange( start = -0.05, stop = 0.05, step = 1e-4 )
plot.plot( xs, np.exp( kd_sklearn_auto.score_samples( np.reshape( xs, ( -1, 1 ) ) ) ), label = 'KDE SKLearn (auto)' )
plot.plot( xs, np.exp( kd_sklearn_manual.score_samples( np.reshape( xs, ( -1, 1 ) ) ) ), label = 'KDE SKLearn (manual)' )
plot.plot( xs, kd_scipy.pdf( xs ), label = 'KDE SciPy' )
plot.hist( data, label = 'Data' )
plot.legend()
plot.show()
```

### Expected Results

Automatic SKLearn bandwidth curve should approximately match SciPy bandwidth curve, roughly the shape of the underlying data histogram.

### Actual Results

Automatic SKLearn bandwidth curve generates a flat PDF.

### Versions

```shell
System:
    python: 3.9.5 (default, Nov 23 2021, 15:27:38)  [GCC 9.3.0]
executable: /local_disk0/.ephemeral_nfs/envs/pythonEnv-67c47d19-3f15-49a2-ab8f-bcf25a2bc29f/bin/python
   machine: Linux-5.15.0-1038-azure-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.2.2
          pip: 21.2.4
   setuptools: 58.0.4
        numpy: 1.20.3
        scipy: 1.7.1
       Cython: 0.29.24
       pandas: 1.3.4
   matplotlib: 3.4.3
       joblib: 1.2.0
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       filepath: /databricks/python3/lib/python3.9/site-packages/numpy.libs/libopenblasp-r0-5bebc122.3.13.dev.so
         prefix: libopenblas
       user_api: blas
   internal_api: openblas
        version: 0.3.13.dev
    num_threads: 6
threading_layer: pthreads
   architecture: Haswell

       filepath: /local_disk0/.ephemeral_nfs/envs/pythonEnv-67c47d19-3f15-49a2-ab8f-bcf25a2bc29f/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
         prefix: libgomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 6

       filepath: /databricks/python3/lib/python3.9/site-packages/scipy.libs/libopenblasp-r0-085ca80a.3.9.so
         prefix: libopenblas
       user_api: blas
   internal_api: openblas
        version: 0.3.9
    num_threads: 6
threading_layer: pthreads
   architecture: Haswell
```

user: kwdwrd-puff
issue_26651: state: open
title: [WIP] FIX Add tests for pyarrow dtypes in pandas Dataframes
body: Co-authored-by: @LeoGrin

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes partly #25896, see also #26464

<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
Adds tests when using the Arrow backend with pandas, for dataframes with arrow dtypes [(added with Pandas 2.0)](https://pandas.pydata.org/pandas-docs/version/2.0/user_guide/pyarrow.html)

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: jovan-stojanovic
issue_26640: state: open
title: ArrayAPI: `isdtype` improper comparison to set
body: ### Describe the bug

The newly introduced `sklearn.utils._array_api.isdtype` is comparing types with respect to defined sets.

See here https://github.com/scikit-learn/scikit-learn/blob/cb233674ddffeb1f75f63309f94e4f90d0a2bdf8/sklearn/utils/_array_api.py#L92

This is not working as NumPy's dtype object is "tricky".

### Steps/Code to Reproduce

```python
>>> np.float32 in {np.float64, np.float32}
False
>>> np.float32 in [np.float64, np.float32]
True
```

Because

```python
>>> np.float32 is np.dtype(np.float32)
False
>>> np.float32 == np.dtype(np.float32)
True
```

### Expected Results

The check should be more robust. I suggest using directly `array_api_compat.xp.isdtype` (or vendoring it).

### Actual Results

N/A

### Versions

```shell
On main
```

cc @thomasjpfan 
user: tupui
issue_26621: state: open
title: Reporting that the EM algorithm didn't converge for a single initialization
body: ### Describe the bug

Should  the code L273-L280
https://github.com/scikit-learn/scikit-learn/blob/364c77e047ca08a95862becf40a04fe9d4cd2c98/sklearn/mixture/_base.py#L273-L280
be indented under the for loop?
https://github.com/scikit-learn/scikit-learn/blob/364c77e047ca08a95862becf40a04fe9d4cd2c98/sklearn/mixture/_base.py#L237

### Steps/Code to Reproduce

Have a look at the bug description.

### Expected Results

We should get a warning for every initialization that fails.

### Actual Results

Currently it does it only for the last initialization as L273-L280 is not indented.

### Versions

```shell
Github.
```

user: krstopro
issue_26617: state: open
title: DOC Introduce dropdowns in the User Guide
body: ### Describe the issue linked to the documentation

Dropdowns are implemented in #26625. They can help users avoid scrolling trough **large** pages and can quickly get them access to the content they are interested in.

### Suggest a potential alternative/fix

Use dropdowns to hide:

- low hierarchy sections such as `References`, `Properties`, etc. See for instance the subsections in [3.3.2.16 Detection error tradeoff (DET)](https://scikit-learn.org/stable/modules/model_evaluation.html#detection-error-tradeoff-det);
- in-depth mathematical details;
- narrative that is too use-case specific;
- narrative that may only interest users that want to go beyond the pragmatics of a given tool.

Additionally:
- Do not use dropdowns for the low level section `Examples`, as it should stay visible to all users. Make sure that the `Examples` section comes right after the main discussion with the least possible folded section in-between.
- Be aware that dropdowns break cross-references. If that makes sense, hide the reference along with the text mentioning it. Else, do not use dropdown.

For more information see [Contributing to documentation](https://scikit-learn.org/stable/developers/contributing.html#documentation), notably the "Guidelines for writing the User Guide and other reStructuredText documents" dropdown.

This is the list of sub-modules to be addressed:

- [x] [1.1. Linear Models](https://scikit-learn.org/stable/modules/linear_model.html) #26623
- [x] [~~1.2. Linear and Quadratic Discriminant Analysis~~](https://scikit-learn.org/stable/modules/lda_qda.html)
- [x] [~~1.3. Kernel ridge regression~~](https://scikit-learn.org/stable/modules/kernel_ridge.html)
- [x] [1.4. Support Vector Machines](https://scikit-learn.org/stable/modules/svm.html) #26641
- [x] [1.5. Stochastic Gradient Descent](https://scikit-learn.org/stable/modules/sgd.html) #26647
- [x] [1.6. Nearest Neighbors](https://scikit-learn.org/stable/modules/neighbors.html)  #27919
- [x] [1.7. Gaussian Processes](https://scikit-learn.org/stable/modules/gaussian_process.html) #27414
- [x] [1.8. Cross decomposition](https://scikit-learn.org/stable/modules/cross_decomposition.html) #27916
- [x] [1.9. Naive Bayes](https://scikit-learn.org/stable/modules/naive_bayes.html) #26819
- [x] [1.10. Decision Trees](https://scikit-learn.org/stable/modules/tree.html) #26699
- [x] [1.11. Ensemble methods](https://scikit-learn.org/stable/modules/ensemble.html) #27915
- [x] [~~1.12. Multiclass and multioutput algorithms~~](https://scikit-learn.org/stable/modules/multiclass.html)
- [x] [1.13. Feature selection](https://scikit-learn.org/stable/modules/feature_selection.html)  #26662
- [x] [~~1.14. Semi-supervised learning~~](https://scikit-learn.org/stable/modules/semi_supervised.html)
- [x] [~~1.15. Isotonic regression~~](https://scikit-learn.org/stable/modules/isotonic.html)
- [x] [~~1.16. Probability calibration~~](https://scikit-learn.org/stable/modules/calibration.html)
- [x] [1.17. Neural network models (supervised)](https://scikit-learn.org/stable/modules/neural_networks_supervised.html) #27920
- [x] [2.1. Gaussian mixture models](https://scikit-learn.org/stable/modules/mixture.html) #26694
- [x] [2.2. Manifold learning](https://scikit-learn.org/stable/modules/manifold.html) #26720
- [x] [2.3. Clustering](https://scikit-learn.org/stable/modules/clustering.html) #26619
- [x] [~~2.4. Biclustering~~](https://scikit-learn.org/stable/modules/biclustering.html)
- [x] [2.5. Decomposing signals in components (matrix factorization problems)](https://scikit-learn.org/stable/modules/decomposition.html) #27551
- [x] [~~2.6. Covariance estimation~~](https://scikit-learn.org/stable/modules/covariance.html)
- [x] [~~2.7. Novelty and Outlier Detection~~](https://scikit-learn.org/stable/modules/outlier_detection.html)
- [x] [2.8. Density Estimation](https://scikit-learn.org/stable/modules/density.html) #26757
- [x] [~~2.9. Neural network models (unsupervised)~~](https://scikit-learn.org/stable/modules/neural_networks_unsupervised.html) #26673
- [x] [3.1. Cross-validation: evaluating estimator performance](https://scikit-learn.org/stable/modules/cross_validation.html) #27921
- [ ] [3.2. Tuning the hyper-parameters of an estimator](https://scikit-learn.org/stable/modules/grid_search.html) #27631
- [x] [3.3. Metrics and scoring: quantifying the quality of predictions](https://scikit-learn.org/stable/modules/model_evaluation.html) #28355
- [x] [~~3.4. Validation curves: plotting scores to evaluate models~~](https://scikit-learn.org/stable/modules/learning_curve.html)
- [x] [4.1. Partial Dependence and Individual Conditional Expectation plots](https://scikit-learn.org/stable/modules/partial_dependence.html) #27187
- [x] [~~4.2. Permutation feature importance~~](https://scikit-learn.org/stable/modules/permutation_importance.html)
- [x] [~~5.1. Available Plotting Utilities~~](https://scikit-learn.org/stable/visualizations.html#available-plotting-utilities)
- [x] [6.1. Pipelines and composite estimators](https://scikit-learn.org/stable/modules/compose.html) #27022
- [x] [6.2. Feature extraction](https://scikit-learn.org/stable/modules/feature_extraction.html) #26807
- [x] [6.3. Preprocessing data](https://scikit-learn.org/stable/modules/preprocessing.html) #27922
- [x] [~~6.4. Imputation of missing values~~](https://scikit-learn.org/stable/modules/impute.html)
- [x] [~~6.5. Unsupervised dimensionality reduction~~](https://scikit-learn.org/stable/modules/unsupervised_reduction.html)
- [x] [~~6.6. Random Projection~~](https://scikit-learn.org/stable/modules/random_projection.html)
- [x] [~~6.7. Kernel Approximation~~](https://scikit-learn.org/stable/modules/kernel_approximation.html)
- [x] [~~6.8. Pairwise metrics, Affinities and Kernels~~](https://scikit-learn.org/stable/modules/metrics.html)
- [x] [~~6.9. Transforming the prediction target (y)~~](https://scikit-learn.org/stable/modules/preprocessing_targets.html)
- [x] [7.1. Toy datasets](https://scikit-learn.org/stable/datasets/toy_dataset.html) #26710
- [x] [7.2. Real world datasets](https://scikit-learn.org/stable/datasets/real_world.html) #26693
- [x] [~~7.3. Generated datasets~~](https://scikit-learn.org/stable/datasets/sample_generators.html)
- [x] [~~7.4. Loading other datasets~~](https://scikit-learn.org/stable/datasets/loading_other_datasets.html)
- [x] [~~8.1. Strategies to scale computationally: bigger data~~](https://scikit-learn.org/stable/computing/scaling_strategies.html)
- [X] [~~8.2. Computational Performance~~](https://scikit-learn.org/stable/computing/computational_performance.html)
- [x] [~~8.3. Parallelism, resource management, and configuration~~](https://scikit-learn.org/stable/computing/parallelism.html)
- [x] [9.1. Python specific serialization](https://scikit-learn.org/stable/model_persistence.html#python-specific-serialization) #26881
- [x] [~~9.2. Interoperable formats~~](https://scikit-learn.org/stable/model_persistence.html#interoperable-formats)
- [x] [~~10.1. Inconsistent preprocessing~~](https://scikit-learn.org/stable/common_pitfalls.html#inconsistent-preprocessing)
- [x] [~~10.2. Data leakage~~](https://scikit-learn.org/stable/common_pitfalls.html#data-leakage)
- [x] [10.3. Controlling randomness](https://scikit-learn.org/stable/common_pitfalls.html#controlling-randomness) #26946

Contributors willing to address this issue, please offer **one of the above sub-modules per pull request**. Remember also to mention on which module you are working on.

Thanks for your help!
user: ArturoAmorQ
labels: good first issue: Easy with clear instructions to resolve
Meta-issue: General issue associated to an identified list of tasks
issue_26614: state: open
title: Column-wise parallelism for Column Transformer
body: ### Describe the workflow you want to enable

cc @Vincent-Maladiere 

Currently, when parallelizing the `ColumnTransformer`, it creates 1 job per transformer. This is inefficient as it may create stragglers.

For instance, in the case when there are 20 columns with a slow transformer (e.g. 1min per col) and 20 others with a very fast transformer (e.g. 1 second per col), parallelizing the operation transformer-wise may take 20mins to run.

On the other hand, if we can parallelize column wise, the operation can go down to 1 min.

### Describe your proposed solution

Parallelize the `ColumnTransformer` on each column, instead of each transformer.

According to @GaelVaroquaux, we could [special case the univariate transformers that can be column-wise parallelized](https://github.com/skrub-data/skrub/issues/586#issuecomment-1586308788) (for instance, [Standard Scaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)).

Multivariate would stay transformer wise parallelized (e.g. [PCA](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html)).

This could be done by introducing transformer tags.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Discussed with the [issue of parallelizing skrub's TableVectorizer](https://github.com/skrub-data/skrub/issues/586).
user: jovan-stojanovic
issue_26595: state: open
title: UX: Enhance the HTML displays
body: ### Describe the workflow you want to enable

When I interact when non-advanced users a recurrent difficulty for them is finding information and understanding what is going on.



### Describe your proposed solution

I think that we can guide users with better html displays.  In general, what would be desirable is to give ways to the users to access all the information that an estimator knows about itself, but avoiding to add any lengthy computation during fit. Of course the difficulty of any UX which is that adding more information leads to crowding, and thus the UX needs to be kept light and focused.

I propose to do changes in an iterative way, adding one feature after the other. Here are the ideas that I have in mind:

* [ ] Display the result of "get_params" (not visible by default, either folded or in a hover)
* [x] Add a link to the API documentation. This link would be inferred from the version of the module, the import path and the name of the class. For instance sklearn.cluster._spectral.SpectralClustering would lead to https://scikit-learn.org/1.2/modules/generated/sklearn.cluster.SpectralClustering.html . Note that we will have to apply heuristics such as dropping the last modules in the path if they are private. Also, we will have to be careful to cater for non scikit-learn classes inheriting from our BaseEstimator, and thus define an override mechanism and probably check that the imported module corresponds to the one for which the path was defined
* [x] Display in a light way whether the estimator has been fit or not
* [ ] Display the feature names

In terms of plan, I propose to first adapt our current display without changing its main philosophy. Hence we need to add light accessors of the information, and not a huge list of things (think "mac", design).

Another important thing to keep in mind is that, for users, the hardest things to comprehend are composite estimators, such as pipelines. Most users do not understand how they can access internal objects in these.

cc @amueller who, if I understand correctly, has been pushing these ideas for a long time. Also cc @thomasjpfan who has always shown impressive skills at html.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

We should make sure that the displays work and are easy to view in all the relevant environments: jupyter notebooks, vscode.

This means that avoid javascript. If needed, we can consider using https://purecss.io/ for buttons, tabs, ...
user: GaelVaroquaux
issue_26586: state: open
title: Array API support for k-nearest neighbors models with the brute force method
body: This issue is a sibling of a similar issue for k-means: #26585 with similar purpose but likely different constraints.

In particular an efficient implementation of k-NN on the GPU would require:

- `torch.cdist`
- `torch.topk` being discussed at:
   - https://github.com/data-apis/array-api/issues/629
user: ogrisel
issue_26585: state: open
title: Array API support for k-means
body: This is an early issue to publicly discuss the possibility (or not) to use the Array API (see #22352) for k-means and make it run on GPUs using PyTorch in particular.

@fcharras has already started to run some promising experiments using the raw PyTorch API. Maybe you could link to a gist with your code?

Unfortunately, the current state of the Array API is likely too limiting because AFAIK it does not yet expose the equivalent of `torch.cdist`, `torch.expand` and `torch.scatter_add_`.

The purpose of this issue is to precisely identify what is blocking us with the current state of Array API and discuss potential solutions:

- use this use case to report to the Array API standardization committee what are our needs to make the spec evolve and benefit everybody;
- alternatively, explore the use of multi-dispatch system such as [uarray](https://github.com/Quansight-Labs/uarray) that is being adopted in scipy to make it possible to maintain a pytorch-specific optimized code path as an alternative to a slower yet generic Array API code path and numpy-optimized code path that would rely on our current Cython code,
- decide that the estimator-level engine API proposed in #25535 is the only sane way to make this estimator GPU (which I now doubt personally).

user: ogrisel
issue_26543: state: open
title: What happend to the idea of adding a 'handle_missing' parameter to the OneHotEncoder?
body: ### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26531

<div type='discussions-op-text'>

<sup>Originally posted by **woodly0** June  7, 2023</sup>
Hello,
I'm having trouble understanding what finally happened to the idea of introducing a `handle_missing` parameter for the `OneHotEncoder`. My current project could still benefit from such an implementation.
There are many existing issues regarding this topic, however, I cannot deduct what was finally decided/implemented and what wasn't.
- #11996
- #12025
- #17317
- #23436

Considering the following features:
```py
import pandas as pd

test_df = pd.DataFrame(
    {"col1": ["red", "blue", "blue"], "col2": ["car", None, "plane"]}
)
```
when using the encoder:
```py
from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(
    handle_unknown="ignore",
    sparse_output=False,
    #handle_missing="ignore"
)
ohe.fit_transform(test_df)
```
I get the output:
```
array([[0., 1., 1., 0., 0.],
       [1., 0., 0., 0., 1.],
       [1., 0., 0., 1., 0.]])
```
but what I'm actually looking for is to remove the `None`, i.e. not create a new feature but set all the others to zero:
```
array([[0., 1., 1., 0.],
       [1., 0., 0., 0.],
       [1., 0., 0., 1.]])
```
Is there a way to achieve this without using another transformer object?</div>
user: woodly0
issue_26530: state: open
title: TransformedTargetRegressor forces 1d y shape to regressor
body: ### Describe the bug

I experience the following error when using TransformedTargetRegressor with my skorch model:
ValueError: The target data shouldn't be 1-dimensional but instead have 2 dimensions, with the second dimension having the same size as the number of regression targets (usually 1). Please reshape your target data to be 2-dimensional (e.g. y = y.reshape(-1, 1).

#### After checking the Source Code this lead me the the following unexpected behaivor which makes little sense:

If TransformedTargetRegressor is fitted with with a 2d dimensional y, it will still be transformed to a 1d dimensional output

y should have the same input and output shapes with a TransformedTargetRegressor or there should be an init argument to disable the change of the input shape
(Yes, internally it gets casted to 2d, but I’m talking about the In and Outputs) 

https://github.com/scikit-learn/scikit-learn/blob/364c77e04/sklearn/compose/_target.py#L20
TransformedTargetRegressor-->fit

```python
        if y.ndim == 1:
            y_2d = y.reshape(-1, 1)
        else:
            y_2d = y
        self._fit_transformer(y_2d)

[...]

        if y_trans.ndim == 2 and y_trans.shape[1] == 1:
            y_trans = y_trans.squeeze(axis=1)
```
But in the end we squeeze it back into a 1d which causes issues for models which expect a 2d input of y
y was 2d in the beginning for a reason

### **The following code would solve this:**
```
        if y_trans.ndim == 2 and y_trans.shape[1] == 1 and y.ndim==1:  #only squeeze back to 1d if y is 1d
            y_trans = y_trans.squeeze(axis=1)
```

This could only create an issue where the y input was for some reason 2d but should be 1d for the regressor. 
In this case an attribute would be nice
```
        if y_trans.ndim == 2 and y_trans.shape[1] == 1 and self.output_dim == 1:
            y_trans = y_trans.squeeze(axis=1)
```

Also in TransformedTargetRegressor-->predict the results dont get squeezed after the prediction of the estimator - only if the original input shape was 1, in that case it is squeezed

So the result looks as expected, but only if the regressor takes a 1d y
If the estimator expects a 2d y the code fails

### Steps/Code to Reproduce

```python
regressor = TransformedTargetRegressor(
    transformer=MinMaxScaler()
)
X, y = np.random.rand(10, 10), np.expand_dims(np.random.rand(10), 1)
regressor.fit(X, y)
```

### Expected Results

The shape of y stays the same as the input OR there is a attribute which allows the choice of  (1d or original)  or (1d or 2d)

input | internal | output
2d —> 2d —> 2d
1d —> 2d —> 1d

### Actual Results

the regressor gets just a 1d array even through y was specifically set to 2d
(I don't know how to extract these results without an debugger)

It works for this example because the default regressor is used, but when using it with other models they might need the 2nd dimention of y, because it was specifically reshaped  (-1,1)

input | internal | output
2d —> 2d —> 1d    THIS creates issues for the regressive which is passed to the Transformer if it expects a 2d array because a 2d y was given 
1d —> 2d —> 1d

### Versions

```shell
System:
    python: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0]
executable: /anaconda/envs/azureml_py310_sdkv2/bin/python
   machine: Linux-5.15.0-1017-azure-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.1.3
          pip: 22.1.2
   setuptools: 61.2.0
        numpy: 1.23.2
        scipy: 1.9.0
       Cython: None
       pandas: 1.4.3
   matplotlib: 3.6.2
       joblib: 1.2.0
threadpoolctl: 3.1.0
```

user: Daniel3009
issue_26505: state: open
title: SLEP006: make sure generated `set_{method}_request` methods are all legit
body: For all estimators inheriting from `BaseEstimator`, if a method explicitly accepts an argument other than `X`, `y`, `Y`, a corresponding `set_{method}_request` is generated for it.

https://github.com/scikit-learn/scikit-learn/pull/26503 proposes the cleanup for `inverse_transform` methods to make sure the corresponding `set_inverse_transform_request` is only generated when necessary.

We should do the same for all other generated methods and make sure they are legit.

A list of all generated methods can be seen using this code:

```py
from sklearn.utils import all_estimators
from sklearn.base import MetaEstimatorMixin
import inspect

for name, Cls in all_estimators():
    is_meta = issubclass(Cls, MetaEstimatorMixin)
    if is_meta:
        print(f"{name} is a meta-estimator")
    else:
        print(name)
    set_methods = [
        name
        for name, _ in inspect.getmembers(Cls, inspect.isroutine)
        if name.startswith("set_") and name.endswith("request")
    ]
    # get input arguments of the methods in set_methods
    args = {
        name: inspect.getfullargspec(getattr(Cls, name)).kwonlyargs
        for name in set_methods
    }
    for key, value in args.items():
        print(f"  {key}: {value}")
    print()
```

Which generates the following output:

<details>

```
ARDRegression
  set_predict_request: ['return_std']
  set_score_request: ['sample_weight']

AdaBoostClassifier is a meta-estimator
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

AdaBoostRegressor is a meta-estimator
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

AdditiveChi2Sampler

AffinityPropagation

AgglomerativeClustering

BaggingClassifier is a meta-estimator
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

BaggingRegressor is a meta-estimator
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

BayesianGaussianMixture

BayesianRidge
  set_fit_request: ['sample_weight']
  set_predict_request: ['return_std']
  set_score_request: ['sample_weight']

BernoulliNB
  set_fit_request: ['sample_weight']
  set_partial_fit_request: ['classes', 'sample_weight']
  set_score_request: ['sample_weight']

BernoulliRBM

Binarizer
  set_transform_request: ['copy']

Birch

BisectingKMeans
  set_fit_request: ['sample_weight']
  set_predict_request: ['sample_weight']
  set_score_request: ['sample_weight']

CCA
  set_predict_request: ['copy']
  set_score_request: ['sample_weight']
  set_transform_request: ['copy']

CalibratedClassifierCV is a meta-estimator
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

CategoricalNB
  set_fit_request: ['sample_weight']
  set_partial_fit_request: ['classes', 'sample_weight']
  set_score_request: ['sample_weight']

ClassifierChain is a meta-estimator
  set_score_request: ['sample_weight']

ColumnTransformer

ComplementNB
  set_fit_request: ['sample_weight']
  set_partial_fit_request: ['classes', 'sample_weight']
  set_score_request: ['sample_weight']

CountVectorizer
  set_fit_request: ['raw_documents']
  set_transform_request: ['raw_documents']

DBSCAN
  set_fit_request: ['sample_weight']

DecisionTreeClassifier
  set_fit_request: ['check_input', 'sample_weight']
  set_predict_proba_request: ['check_input']
  set_predict_request: ['check_input']
  set_score_request: ['sample_weight']

DecisionTreeRegressor
  set_fit_request: ['check_input', 'sample_weight']
  set_predict_request: ['check_input']
  set_score_request: ['sample_weight']

DictVectorizer
  set_inverse_transform_request: ['dict_type']

DictionaryLearning

DummyClassifier
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

DummyRegressor
  set_fit_request: ['sample_weight']
  set_predict_request: ['return_std']
  set_score_request: ['sample_weight']

ElasticNet
  set_fit_request: ['check_input', 'sample_weight']
  set_score_request: ['sample_weight']

ElasticNetCV
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

EllipticEnvelope
  set_score_request: ['sample_weight']

EmpiricalCovariance
  set_score_request: ['X_test']

ExtraTreeClassifier
  set_fit_request: ['check_input', 'sample_weight']
  set_predict_proba_request: ['check_input']
  set_predict_request: ['check_input']
  set_score_request: ['sample_weight']

ExtraTreeRegressor
  set_fit_request: ['check_input', 'sample_weight']
  set_predict_request: ['check_input']
  set_score_request: ['sample_weight']

ExtraTreesClassifier is a meta-estimator
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

ExtraTreesRegressor is a meta-estimator
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

FactorAnalysis

FastICA
  set_inverse_transform_request: ['copy']
  set_transform_request: ['copy']

FeatureAgglomeration
  set_inverse_transform_request: ['Xred']

FeatureHasher
  set_transform_request: ['raw_X']

FeatureUnion

FunctionTransformer

GammaRegressor
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

GaussianMixture

GaussianNB
  set_fit_request: ['sample_weight']
  set_partial_fit_request: ['classes', 'sample_weight']
  set_score_request: ['sample_weight']

GaussianProcessClassifier
  set_score_request: ['sample_weight']

GaussianProcessRegressor
  set_predict_request: ['return_cov', 'return_std']
  set_score_request: ['sample_weight']

GaussianRandomProjection

GenericUnivariateSelect

GradientBoostingClassifier is a meta-estimator
  set_fit_request: ['monitor', 'sample_weight']
  set_score_request: ['sample_weight']

GradientBoostingRegressor is a meta-estimator
  set_fit_request: ['monitor', 'sample_weight']
  set_score_request: ['sample_weight']

GraphicalLasso
  set_score_request: ['X_test']

GraphicalLassoCV
  set_score_request: ['X_test']

GridSearchCV is a meta-estimator
  set_fit_request: ['groups']
  set_inverse_transform_request: ['Xt']

HDBSCAN

HashingVectorizer

HistGradientBoostingClassifier
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

HistGradientBoostingRegressor
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

HuberRegressor
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

IncrementalPCA
  set_partial_fit_request: ['check_input']

IsolationForest is a meta-estimator
  set_fit_request: ['sample_weight']

Isomap

IsotonicRegression
  set_fit_request: ['sample_weight']
  set_predict_request: ['T']
  set_score_request: ['sample_weight']
  set_transform_request: ['T']

KBinsDiscretizer
  set_fit_request: ['sample_weight']
  set_inverse_transform_request: ['Xt']

KMeans
  set_fit_request: ['sample_weight']
  set_predict_request: ['sample_weight']
  set_score_request: ['sample_weight']

KNNImputer

KNeighborsClassifier
  set_score_request: ['sample_weight']

KNeighborsRegressor
  set_score_request: ['sample_weight']

KNeighborsTransformer

KernelCenterer
  set_fit_request: ['K']
  set_transform_request: ['K', 'copy']

KernelDensity
  set_fit_request: ['sample_weight']

KernelPCA

KernelRidge
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

LabelBinarizer
  set_inverse_transform_request: ['threshold']

LabelEncoder

LabelPropagation
  set_score_request: ['sample_weight']

LabelSpreading
  set_score_request: ['sample_weight']

Lars
  set_fit_request: ['Xy']
  set_score_request: ['sample_weight']

LarsCV
  set_fit_request: ['Xy']
  set_score_request: ['sample_weight']

Lasso
  set_fit_request: ['check_input', 'sample_weight']
  set_score_request: ['sample_weight']

LassoCV
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

LassoLars
  set_fit_request: ['Xy']
  set_score_request: ['sample_weight']

LassoLarsCV
  set_fit_request: ['Xy']
  set_score_request: ['sample_weight']

LassoLarsIC
  set_fit_request: ['copy_X']
  set_score_request: ['sample_weight']

LatentDirichletAllocation

LedoitWolf
  set_score_request: ['X_test']

LinearDiscriminantAnalysis
  set_score_request: ['sample_weight']

LinearRegression
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

LinearSVC
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

LinearSVR
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

LocalOutlierFactor

LocallyLinearEmbedding

LogisticRegression
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

LogisticRegressionCV
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

MDS
  set_fit_request: ['init']

MLPClassifier
  set_partial_fit_request: ['classes']
  set_score_request: ['sample_weight']

MLPRegressor
  set_score_request: ['sample_weight']

MaxAbsScaler

MeanShift

MinCovDet
  set_score_request: ['X_test']

MinMaxScaler

MiniBatchDictionaryLearning

MiniBatchKMeans
  set_fit_request: ['sample_weight']
  set_partial_fit_request: ['sample_weight']
  set_predict_request: ['sample_weight']
  set_score_request: ['sample_weight']

MiniBatchNMF
  set_inverse_transform_request: ['W']
  set_partial_fit_request: ['H', 'W']

MiniBatchSparsePCA

MissingIndicator

MultiLabelBinarizer
  set_inverse_transform_request: ['yt']

MultiOutputClassifier is a meta-estimator
  set_fit_request: ['sample_weight']
  set_partial_fit_request: ['classes', 'sample_weight']

MultiOutputRegressor is a meta-estimator
  set_fit_request: ['sample_weight']
  set_partial_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

MultiTaskElasticNet
  set_fit_request: ['check_input', 'sample_weight']
  set_score_request: ['sample_weight']

MultiTaskElasticNetCV
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

MultiTaskLasso
  set_fit_request: ['check_input', 'sample_weight']
  set_score_request: ['sample_weight']

MultiTaskLassoCV
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

MultinomialNB
  set_fit_request: ['sample_weight']
  set_partial_fit_request: ['classes', 'sample_weight']
  set_score_request: ['sample_weight']

NMF
  set_inverse_transform_request: ['W']

NearestCentroid
  set_score_request: ['sample_weight']

NearestNeighbors

NeighborhoodComponentsAnalysis

Normalizer
  set_transform_request: ['copy']

NuSVC
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

NuSVR
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

Nystroem

OAS
  set_score_request: ['X_test']

OPTICS

OneClassSVM
  set_fit_request: ['sample_weight']

OneHotEncoder

OneVsOneClassifier is a meta-estimator
  set_partial_fit_request: ['classes']
  set_score_request: ['sample_weight']

OneVsRestClassifier is a meta-estimator
  set_partial_fit_request: ['classes']
  set_score_request: ['sample_weight']

OrdinalEncoder

OrthogonalMatchingPursuit
  set_score_request: ['sample_weight']

OrthogonalMatchingPursuitCV
  set_score_request: ['sample_weight']

OutputCodeClassifier is a meta-estimator
  set_score_request: ['sample_weight']

PCA

PLSCanonical
  set_predict_request: ['copy']
  set_score_request: ['sample_weight']
  set_transform_request: ['copy']

PLSRegression
  set_predict_request: ['copy']
  set_score_request: ['sample_weight']
  set_transform_request: ['copy']

PLSSVD

PassiveAggressiveClassifier
  set_fit_request: ['coef_init', 'intercept_init']
  set_partial_fit_request: ['classes']
  set_score_request: ['sample_weight']

PassiveAggressiveRegressor
  set_fit_request: ['coef_init', 'intercept_init']
  set_partial_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

PatchExtractor

Perceptron
  set_fit_request: ['coef_init', 'intercept_init', 'sample_weight']
  set_partial_fit_request: ['classes', 'sample_weight']
  set_score_request: ['sample_weight']

Pipeline
  set_inverse_transform_request: ['Xt']
  set_score_request: ['sample_weight']

PoissonRegressor
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

PolynomialCountSketch

PolynomialFeatures

PowerTransformer

QuadraticDiscriminantAnalysis
  set_score_request: ['sample_weight']

QuantileRegressor
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

QuantileTransformer

RANSACRegressor is a meta-estimator
  set_fit_request: ['sample_weight']

RBFSampler

RFE is a meta-estimator

RFECV is a meta-estimator
  set_fit_request: ['groups']

RadiusNeighborsClassifier
  set_score_request: ['sample_weight']

RadiusNeighborsRegressor
  set_score_request: ['sample_weight']

RadiusNeighborsTransformer

RandomForestClassifier is a meta-estimator
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

RandomForestRegressor is a meta-estimator
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

RandomTreesEmbedding is a meta-estimator
  set_fit_request: ['sample_weight']

RandomizedSearchCV is a meta-estimator
  set_fit_request: ['groups']
  set_inverse_transform_request: ['Xt']

RegressorChain is a meta-estimator
  set_score_request: ['sample_weight']

Ridge
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

RidgeCV
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

RidgeClassifier
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

RidgeClassifierCV
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

RobustScaler

SGDClassifier
  set_fit_request: ['coef_init', 'intercept_init', 'sample_weight']
  set_partial_fit_request: ['classes', 'sample_weight']
  set_score_request: ['sample_weight']

SGDOneClassSVM
  set_fit_request: ['coef_init', 'offset_init', 'sample_weight']
  set_partial_fit_request: ['sample_weight']

SGDRegressor
  set_fit_request: ['coef_init', 'intercept_init', 'sample_weight']
  set_partial_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

SVC
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

SVR
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

SelectFdr

SelectFpr

SelectFromModel is a meta-estimator

SelectFwe

SelectKBest

SelectPercentile

SelfTrainingClassifier is a meta-estimator

SequentialFeatureSelector is a meta-estimator

ShrunkCovariance
  set_score_request: ['X_test']

SimpleImputer

SkewedChi2Sampler

SparseCoder

SparsePCA

SparseRandomProjection

SpectralBiclustering

SpectralClustering

SpectralCoclustering

SpectralEmbedding

SplineTransformer
  set_fit_request: ['sample_weight']

StackingClassifier is a meta-estimator
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

StackingRegressor is a meta-estimator
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

StandardScaler
  set_fit_request: ['sample_weight']
  set_inverse_transform_request: ['copy']
  set_partial_fit_request: ['sample_weight']
  set_transform_request: ['copy']

TSNE

TargetEncoder

TfidfTransformer
  set_transform_request: ['copy']

TfidfVectorizer
  set_fit_request: ['raw_documents']
  set_transform_request: ['raw_documents']

TheilSenRegressor
  set_score_request: ['sample_weight']

TransformedTargetRegressor
  set_score_request: ['sample_weight']

TruncatedSVD

TweedieRegressor
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

VarianceThreshold

VotingClassifier is a meta-estimator
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']

VotingRegressor is a meta-estimator
  set_fit_request: ['sample_weight']
  set_score_request: ['sample_weight']
```

</details>

Observations, among other things:

- `sample_weight` for non-meta estimators is legit, but for meta-estimators needs checking. Not all meta-estimators actually consume `sample_weight`, but many of them have it as an explicit arg.
- Not all meta-estimators inherit from `MetaEstimatorMixin`, therefore not marked as meta-estimator above.
- `copy` should probably be considered metadata, but not sure
- `check_input` should probably not be considered metadata
- `return_std` should probably be considered metadata
user: adrinjalali
issue_26502: state: open
title: [Bug] Wrong grad computation for RBF Gaussian Process Regression
body: I think there might be a wrong grad computation in RBF Gaussain Process Kenerl, or I misread the codes here:
https://github.com/scikit-learn/scikit-learn/blob/4509e5810be8bafa0199fabba5691516d36f02c2/sklearn/gaussian_process/kernels.py#L1551-L1553
My understanding is `K_gradient` is the gradient of  RBF Kernel ( `K`) with respect to `lenght_scale` (L). The Expression of `K` (without constant Kernel) is shwon below:
```math
K(r) = exp\left(- 0.5* \left(  \frac{r}{L} \right)^2 \right) 
```
Meanwhile, a pixel-wise gradient of `K` with respect to `L` is:
```math
\frac{\partial K(r)}{\partial L} = exp\left(- 0.5* \left(  \frac{r}{L} \right)^2 \right) \cdot \left( r^2 L^{-3} \right)
```
However, in the code, the pixel-wise gradient is
```math
\frac{\partial K(r)}{\partial L} = exp\left(- 0.5* \left(  \frac{r}{L} \right)^2 \right) \cdot \left( r^2 L^{-2} \right)
```
I think the exponent of `L` (lenght scale) in `K_gradient` is **-3** rather than **-2**.
I appreciate your reply.
user: gitouni
issue_26494: state: open
title: Return training loss from LogisticRegression
body: ### Describe the workflow you want to enable

Currently there seems to be no way to retrieve the loss (or change in loss) when training a logistic regression model (`sklearn.linear_model.LogisticRegression`). If one changes the verbosity then this can be seen from the terminal, but also cannot be retrieved from stdout. This makes it unfeasible to actually plot training curves from logistic regression models. 

### Describe your proposed solution

Add `train_loss` as an attribute, same as e.g. `classes_` or `n_features_in_`.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: rachrapp
issue_26493: state: open
title: Inconsitency between C-contiguous and F-contiguous arrays
body: ### No consistency between C-contiguous and F-contiguous arrays for LinearRegression()

At least for LinearRegression() : In some edge case (when X is almost singular), there is huge difference between C-contiguous and F-contiguous arrays predictions.

- This is due to the fact that array product gives different results between contiguous and F-contiguous arrays ([cf this Stack Overflow questions that I posted](https://stackoverflow.com/questions/76388886/python-rounding-errors-between-c-contiguous-and-f-contiguous-arrays-for-matrix))
- These "edge cases" can actually be quite common in time-series predictions, where a lot of auto-regressive features can easily be correlated
- I would strongly advise parsing all arrays to C-contiguous before doing the predictions/fitting.
- Please also note that **fitting** with F-contiguous or C-contiguous can also give different results.
- The worst is not that this is happening, it is that no warning are being raised whatsoever.
- Also, F-contiguous arrays are extremely common in pandas DataFrames, which is what a lot of developers are using in this context...

### Steps/Code to Reproduce

```python
import numpy as np; print(np.__version__) # 1.23.5
import scipy; print(scipy.__version__) # 1.10.0
import sklearn as sk; print(sk.__version__) # 1.2.1

from sklearn.linear_model import LinearRegression
import pandas as pd

# Parameters 
seed, N_obs, N_feat, mu_x, sigma_x, mu_y, sigma_y = 0, 100, 1000, 100, 0.1, 100, 1

# 1) Creating a weird edge-case X, y :
np.random.seed(seed)
s = pd.Series(np.random.normal(mu_x, sigma_x, N_obs))
X = np.stack([s.ewm(com=com).mean() for com in np.arange(N_feat)]).T
y = np.random.normal(mu_y, sigma_y, N_obs)

# 2) Showing that there is different results for C-cont vs F-cont arrays :
model = LinearRegression()
model.fit(X, y)
y_pred = model.predict(X)
y_pred_c = model.predict(np.ascontiguousarray(X))

# Either just plot it and see :
import matplotlib.pyplot as plt
plt.scatter(y_pred, y_pred_c)

# Or look at the data :
np.var(y_pred)
np.var(y_pred - y_pred_c)
np.corrcoef(y_pred, y_pred_c)[0,1] # == 0.40295584536349216
# --> y_pred EXTREMELY different than y_pred_c
```

### Expected Results

We expect y_pred to be fully equal to y_pred_c.
Or at least `np.corrcoef(y_pred, y_pred_c)[0,1] > .99`


### Actual Results

`np.corrcoef(y_pred, y_pred_c)[0,1] # == 0.40295584536349216`
- ypred and y_pred_c are totally different.

### Versions

```shell
System:
    python: 3.10.9 (main, Mar  1 2023, 18:23:06) [GCC 11.2.0]
executable: /opt/anaconda3/bin/python
   machine: Linux-5.10.0-23-cloud-amd64-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.2.1
          pip: 22.3.1
   setuptools: 65.6.3
        numpy: 1.23.5
        scipy: 1.10.0
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.7.0
       joblib: 1.1.1
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       filepath: /opt/anaconda3/lib/libmkl_rt.so.1
         prefix: libmkl_rt
       user_api: blas
   internal_api: mkl
        version: 2021.4-Product
    num_threads: 64
threading_layer: intel

       filepath: /opt/anaconda3/lib/libiomp5.so
         prefix: libiomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 128

       filepath: /opt/anaconda3/lib/libgomp.so.1.0.0
         prefix: libgomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 128
```

user: JeanLescutMuller
issue_26491: state: open
title: Rename percentiles to quantile_levels in partial_dependence
body: `partial_dependence` and `PartialDependenceDisplay` have a parameter `percentiles` with range (0, 1). It should be either have range (0, 100) or be named `quantiles`.

This issue proposes to rename `percentiles` to ~~`quantiles`~~ `quantile_levels`.

This is a similar proposal as in #26090.
user: lorentzenchr
labels: Needs Decision - API: Requires decision regarding API
issue_26482: state: open
title: Run common test for SparseCoder and FeatureUnion
body: Currently, the `SparseCoder` estimator is not tested by our common test because it requires a `dictionary` parameter.

We should make sure to construct the instance in `_construct_instance` and check that the estimator runs the test. I am almost sure that it is missing the `"stateless"` tag and the parameter validation.
user: glemaitre
labels: Moderate: Anything that requires some knowledge of conventions and best practices
module:test-suite: everything related to our tests
issue_26480: state: open
title: FIX Backwards `SequentialFeatureSelector` always drops one feature
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Closes #26369 

#### What does this implement/fix? Explain your changes.

This is an attempt at closing the issue. It uses the "all features selected" cross val score as the starting value instead of `-inf` for backwards selection. ~~I also changed the forward selection starting value to `0`, I think scores have to be between zero and one?!~~

I also had to update the tests after this change, so I am not sure the changes are correct. (Maybe the test was wrong, but maybe not?)

This is just a draft/way to store this work in progress somewhere and lets others see it. Maybe someone wants to take on this PR.

#### Any other comments?

There is code to reproduce the problem in the original issue. It could be a good starting point for making a non-regression test.
<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: betatim
issue_26462: state: open
title: [MRG] TST use global_random_seed in sklearn/utils/tests/test_optimize.py
body: Towards #22827

I modified the only test in [sklearn/utils/tests/test_optimize.py](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/tests/test_optimize.py) to use the `global_random_seed` fixture.

Required changes were minimal:
Just replaced the previously fixed seed with `global_random_seed` and pass the fixture to the test function.

All tests run via 
```
SKLEARN_TESTS_GLOBAL_RANDOM_SEED="all" pytest sklearn/utils/tests/test_optimize.py
```
pass locally.

The PR only changes a test file, so no addition to the change log required?
user: sply88
issue_26460: state: open
title: Pandas set_output Development and Customer Transformers Best Practice
body: ### Describe the issue linked to the documentation

The set_output feature is amazing. However, many of my pipelines have sklearn style custom transformers. It would be really useful to have some best practices on how to write custom transformer objects so that they maintain the functionality of the set_output API.

The closest I have found is this: https://stackoverflow.com/questions/75026592/how-to-create-pandas-output-for-custom-transformers, but while an excellent answer on stackoverflow, it would be great to see some offical best practices on it. I think it would be a really useful for more advanced pipelines.

Many thanks. Awesome work 🚀 

### Suggest a potential alternative/fix

_No response_
user: joshdunnlime
issue_26437: state: open
title: ENH allow make_spd_matrix to generate multiple matrices at once
body: #### Reference Issues/PRs

None

#### What does this implement/fix? Explain your changes.

This PR aims to use NumPy broadcasting to generate several matrices simultaneously, adding parameter `n_samples`.
Computational time is unchanged for `n_samples`=1.
user: qbarthelemy
issue_26430: state: open
title: Conformal inference
body: ### Describe the workflow you want to enable

Conformal prediction (CP) is a statistical technique for producing prediction sets without assumptions on the predictive algorithm (often a machine learning system) and only assuming exchangeability of the data. Given this method is assumption lean and thereby very defendable for causal inference, it would be nice if there was out-of-the-box support. 

### Describe your proposed solution

Some example code can be found [here](https://github.com/aangelopoulos/conformal-prediction). There would need to be at least 2 implementations:
1. Categorical: https://github.com/aangelopoulos/conformal-prediction/blob/main/notebooks/imagenet-aps-randomized.ipynb
2. Numeric: https://github.com/aangelopoulos/conformal-prediction/blob/main/notebooks/meps-cqr.ipynb
[Seminal paper that proves coverage (2017)](https://www.stat.cmu.edu/~ryantibs/papers/conformal.pdf)

### Describe alternatives you've considered, if relevant

The current options for using conformal prediction with an sklearn model are:
1. Write your own code
2. Import from [MAPIE](https://mapie.readthedocs.io/en/latest/index.html) or another library

### Additional context

I'm happy to implement these changes, but before filing a PR I wanted to get feedback if this aligns with sklearn's structure. 
user: michael-berk
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_26428: state: open
title: `HistGradientBoosting` should shuffle features when exploring for new splits
body: ### Summary

Our implementation of `HistGradientBoosting` does not shuffle the feature at each node to find the best split. Note that our `GradientBoosting`, `RandomForest`, and `DecisionTree` use a Fisher-Yates permutation.

Not permuting features introduces a bias that can have some impact. For instance, it will impact the inspection and could lead to misinterpretation if one does not know about this implementation detail.

Below, I show an example where correlated features will have close importance in the case of a `GradientBoosting`. For `HistGradientBoosting`, a single feature (in an arbitrary manner) will be used and thus only a single feature is reported as important. 

### Reproducer

```python
# %%
import numpy as np

rng = np.random.default_rng(42)

n_samples, n_features = 100, 3
X = rng.normal(size=(n_samples, n_features))
X = np.concatenate([X, X], axis=1)  # create correlated features
y = X[:, 0] + X[:, 1] + rng.normal(scale=0.1, size=n_samples)

# %%
from sklearn.ensemble import GradientBoostingRegressor, HistGradientBoostingRegressor

gbdt = GradientBoostingRegressor(n_estimators=100, random_state=42).fit(X, y)
hgbdt = HistGradientBoostingRegressor(max_iter=100, random_state=42).fit(X, y)

# %%
import pandas as pd
from sklearn.inspection import permutation_importance

importances = {}
for name, model in zip(["gbdt", "hgbdt"], [gbdt, hgbdt]):
    result = permutation_importance(model, X, y, n_repeats=100, random_state=42, n_jobs=-1)
    importances[name] = pd.DataFrame(
        result.importances.T, columns=[f"Features #{i}" for i in range(X.shape[1])]
    )

# %%
import matplotlib.pyplot as plt

fig, axes = plt.subplots(ncols=2, figsize=(10, 4))
for ax, (name, importance) in zip(axes, importances.items()):
    importance.plot.box(ax=ax, vert=False, whis=100)
    ax.set_title(name)
    ax.set_xlabel("Decrease in R2 score")
fig.tight_layout()
```

![image](https://github.com/scikit-learn/scikit-learn/assets/7454015/acd0956f-fc87-4df9-bc3a-448becb108d3)

### Resolution

We can use a shuffling in `_find_node_split` to alleviate this bias and have a similar implementation as in the current tree implementation.

user: glemaitre
issue_26423: state: open
title: FEA add stratified k-fold iterators for splitting multilabel data
body: #### Reference Issues/PRs

Towards #25193.

#### What does this implement/fix? Explain your changes.

This PR adds support for `"multilabel-indicator"` in `StratifiedKFold`, which implements the iterative stratification algorithm for multi-label classification (I partially referred to the implementation [here](https://github.com/trent-b/iterative-stratification/blob/master/iterstrat/ml_stratifiers.py#L109)). Correspondingly, `RepeatedStratifiedKFold` would now support `"multilabel-indicator"` as well. The paper that proposed iterative stratification is [this](http://lpis.csd.auth.gr/publications/sechidis-ecmlpkdd-2011.pdf) and [this video](https://videolectures.net/ecmlpkdd2011_tsoumakas_stratification/?q=stratification%20multi%20label) may be helpful for understanding the algorithm.

`StratifiedKFold` with multi-label target is tested in the following aspects (except the basics):
- It preserves data ordering as much as possible.
- It preserves the ratio of positive to negative examples of each label in individual splits.
- The difference between maximum and minimum test sizes is at most 1.
- The stratification gives same indices regardless of the labels actually are.
- Shuffling happens when requested, as for single-label target.

As for documentation:
- This PR includes a brief example in `modules/doc/cross_validation.rst`.
- I have drafted a visualization [here](https://gist.github.com/Charlie-XIAO/75930931572afae389c29a7255947fca) based on `examples\model_selection\plot_cv_indices.py`, though it is not implemented in the PR yet.

Some other comments:
- `[["0", "1"], ["1", "0"]]` is considered `"multiclass-multioutput"` instead of `"multiclass-indicator"`. Is this the desired behavior?
- I'm thinking that `model_selection/tests/test_split.py` may need to be refactored... It seems too messy now.
user: Charlie-XIAO
issue_26418: state: open
title: RFC Supporting `scipy.sparse.sparray`
body: ### Context

SciPy is now favoring sparse arrays (i.e. `scipy.sparse.sparray` and its subclasses) over sparse matrices (i.e. `scipy.sparse.spmatrix` and its subclasses) to enlarge the scope of matrices to $n$-dimensional data-structures since SciPy 1.8 (see https://github.com/scipy/scipy/pull/14822).

Sparse matrices now subclass their associated sparse arrays (see https://github.com/scipy/scipy/pull/18440).

scikit-learn has been supporting sparse matrices but now also needs to support SciPy sparse arrays.

### Proposed solutions

Ordered by preference:
 - Use `scipy.sparse.issparse` and the `format` attribute everywhere and not use `isinstance` at all
 - Use `isinstance` on private compatibility class defined to be `scipy.sparse.spmatrix` or `scipy.sparse.sparray` conditionally on SciPy's version (i.e. use `scipy.sparse.sparray` if available)
 - Rely on duck-typing or the class name to check for sparse arrays

cc @ivirshup
user: jjerphan
labels: Needs Decision - API: Requires decision regarding API
issue_26403: state: open
title: [MRG] TST use global_random_seed in sklearn/decomposition/tests/test_pca.py
body: Towards #22827

I modified the relevant tests in [sklearn/decomposition/tests/test_pca.py](https://github.com/scikit-learn/scikit-learn/blob/364c77e047ca08a95862becf40a04fe9d4cd2c98/sklearn/decomposition/tests/test_pca.py) to use the `global_random_seed` fixture.
Tests that relate to shapes, dtypes, exceptions and warnings were not modified.

All tests run via
```
SKLEARN_TESTS_GLOBAL_RANDOM_SEED="all" pytest sklearn/decomposition/tests/test_pca.py
```
pass locally.

While most tests only required minimal adjustments, some tests did require slightly changing datasets or test parameterization. I added comments for the respective code sections below.

The PR only changes a test file, so no addition to change log required?
user: sply88
issue_26401: state: open
title: Numpy Array Error when Training LogisticRegressionCV
body: ### Describe the bug

When I attempt to train LogisticRegressionCV, I get the error: setting an array element with a sequence. The requested array has an inhomogeneous shape after 2 dimensions. The detected shape was (5, 10) + inhomogeneous part.

The inputs to LogisticRegressionCV are:
- X = output of StandardScaler.fit_transform, with a shape of (24, 12) and dtype of float64
- y = array with shape (24, ) and dtype of int64

I checked that there are not null or infinite values in either array.

### Steps/Code to Reproduce

```
data_for_color_training = training_data.loc[training_data["IdentCode"] == color]
x_train = data_for_color_training.loc[:, (data_for_color_training.columns.str.startswith("dx") | data_for_color_training.columns.str.startswith("dy") | data_for_color_training.columns.str.startswith("dz"))]
y_train = data_for_color_training["Grade"]
			
#--------------Edit specific pre-processing and model here-----------------------------
scaler = StandardScaler()
X_train = scaler.fit_transform(x_train)
modelTrained = LogisticRegressionCV(max_iter=10000).fit(X_train, y_train)
```

### Expected Results

The variable modelTrained will output a LogisticRegressionCV model.

### Actual Results
```traceback
Traceback (most recent call last):
  File "C:\Users\...\PythonScripts\UserDefinedModel.py", line 22, in train_predict_by_colorcode
    modelTrained = LogisticRegressionCV(max_iter=10000).fit(X_train, y_train)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\...\AppData\Local\Programs\Python\Python311\Lib\site-packages\sklearn\linear_model\_logistic.py", line 1912, in fit
    coefs_paths = np.reshape(
                  ^^^^^^^^^^^
  File "<__array_function__ internals>", line 200, in reshape
  File "C:\Users\...\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\core\fromnumeric.py", line 298, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\...\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\core\fromnumeric.py", line 54, in _wrapfunc
    return _wrapit(obj, method, *args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\...\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\core\fromnumeric.py", line 43, in _wrapit
    result = getattr(asarray(obj), method)(*args, **kwds)
                     ^^^^^^^^^^^^
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 2 dimensions. The detected shape was (5, 10) + inhomogeneous part.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\...\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\core\fromnumeric.py", line 298, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\...\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\core\fromnumeric.py", line 54, in _wrapfunc
    return _wrapit(obj, method, *args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\...\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\core\fromnumeric.py", line 43, in _wrapit
    result = getattr(asarray(obj), method)(*args, **kwds)
                     ^^^^^^^^^^^^
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 2 dimensions. The detected shape was (5, 10) + inhomogeneous part.
```
### Versions

```shell
System:
    python: 3.11.1 (tags/v3.11.1:a7a450f, Dec  6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)]
executable: C:\Users\...\AppData\Local\Programs\Python\Python311\python.exe
   machine: Windows-10-10.0.19044-SP0

Python dependencies:
      sklearn: 1.2.2
          pip: 22.3.1
   setuptools: 65.5.0
        numpy: 1.24.1
        scipy: 1.10.0
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.6.3
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\...\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
        version: 0.3.21
threading_layer: pthreads
   architecture: Haswell
    num_threads: 16

       user_api: openmp
internal_api: openmp
         prefix: vcomp
       filepath: C:\Users\...\AppData\Local\Programs\Python\Python311\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 16

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\...\AppData\Local\Programs\Python\Python311\Lib\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell
    num_threads: 16
```

user: ajr426
issue_26395: state: open
title: GridSearchCV support callback for MLFlow 
body: ### Describe the workflow you want to enable

I would like to save off the results of all runs in GridSearchCV to MLFlow.  MLFlow 

```python
for param in params:
    with mlflow.start_run():
        est = ElasticNet(**param)
        est.fit(train_x, train_y)
        metrics = est.score(test_x, test_y)
        mlflow.log_params(param)
        mlflow.log_metrics(metrics)
        mlflow.sklearn.log_model(est, "model")
```

See https://mlflow.org/docs/latest/tutorials-and-examples/tutorial.html for more details:

I would like to use `GridSearchCV` to do the above because it comes with many other features (e.g. `HalvingGridSearchCV`, multi-threading, etc ...)[](url)

### Describe your proposed solution

A callback parameter to `GridSearchCV`.  Perhaps
```python
def log_candidate(model, test_x, test_y):
  with mlflow.start_run():
        mlflow.log_params(model.get_params())
        mlflow.log_metrics(metrics)
        mlflow.sklearn.log_model(est, "model")
```

### Describe alternatives you've considered, if relevant

To hack the scorer for this purpose: https://danielhnyk.cz/adding-callback-to-a-sklearn-gridsearch/

This is suboptimal because:
1. If you want to return multiple metrics, you cannot save multiple scores using the provided API.  This is because we have to pass multiple **scorer**s, not a function that generates multiple scores.
2. Enabling `return_train_score` will call the scorer callback too many times and it is not easy to distinguish between the training and testing scoring.

### Additional context

_No response_
user: tianhuil
issue_26390: state: open
title: SplineTransformer(extrapolate="periodic") outputs nan values for constant features
body: While reviewing #24145 I discovered the following bug:

```python
In [1]: import numpy as np

In [2]: from sklearn.preprocessing import SplineTransformer

In [3]: SplineTransformer(extrapolation="periodic").fit_transform(np.ones(shape=(5, 1)))
Out[3]: 
array([[nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan]])
```

Batman!

This is caused by:

- https://github.com/scikit-learn/scikit-learn/blob/ea046f024694b9a558c882b8c2610c52dad95e29/sklearn/preprocessing/_polynomial.py#L962-L971

`spl.t[n] - spl.t[spl.k]` is typically zero for constant features.

While fixing it for constant features is probably easy (just use `x = X[:, i]`), I wonder if other related numerical stability problems can be triggered for nearly constant features.

But maybe we can solve this problem in two stages: first the nan problem caused by modulus by exact zero and then investigate behavior on nearly constant data.

user: ogrisel
issue_26369: state: open
title: SequentialFeatureSelector in backward auto mode will always remove one feature
body: https://github.com/scikit-learn/scikit-learn/blob/364c77e047ca08a95862becf40a04fe9d4cd2c98/sklearn/feature_selection/_sequential.py#L273

The initial value of `old_score` is incorrect if `direction == 'backward'`. With the current initial value of `-np.inf, ((new_score - old_score) < self.tol)` would always be `False` no matter what `new_score` is returned by `self._get_best_new_feature_score()`, so that at least one feature would be removed in the first iteration.

`old_score` needs to be set to an initial value that is the `cross_val_score` with all the features in use if `direction == 'backward'`.
user: sktin
issue_26367: state: open
title: ENH `despine` keyword for ROC and PR curves
body: #### Reference Issues/PRs

Towards #25929.

#### What does this implement/fix? Explain your changes.

This PR adds a new keyword `despine` to remove the top and right axes in order to make the plot clearer. For examples, please see this gist: https://gist.github.com/Charlie-XIAO/ecd5173fe0daa9c49f442153ef19e7d1

#### Any other comments?

This PR need to have PR #26366 and PR #26368 merged in advance. Also, this is related and may have a lot of conflicts with PR #26019.

user: Charlie-XIAO
issue_26365: state: open
title: DOC improve example by reusing best_estimator_
body: related #23243 

done:
- have made modification to use the best_estimator_ for plots as suggested in the first bullet of the related issue.

to do:
- [ ] will add additional description of the code for the first section (creating dataset, construction of the kernel model and time comparison) to give more context.
user: dromeropa
issue_26364: state: open
title: Adding dark mode for pipeline diagram
body: ### Describe the workflow you want to enable

The current diagram truly hurts eyes when everything else in dark mode. It would be a very nice feature to add.
![Screenshot 2023-05-14 100107](https://github.com/scikit-learn/scikit-learn/assets/95470962/bf9d7275-be3a-4864-82ce-362a7566d22e)


### Describe your proposed solution


Currently I do it this way, but I bet there could have been a better way.

`
def display_dark(pipeline, text_color="#3bd5eb"):
    replacements = {
        "color: black": f"color: {text_color};",
        "background-color: white": "background-color: transparent",
        'class="sk-toggleable__label sk-toggleable__label-arrow"': 'class="sk-toggleable__label sk-toggleable__label-arrow" style="background-color: #222831;"',
        "#696969;": "#e6e6e6;",
        "#d4ebff;": "#1a1a1a;",
        "background-color: #f0f8ff;": "background-color: #393e46",
        "background-color: #d4ebff;": "background-color: #1a1a1a;",
        "color: #e6e6e6;": "color: #b3b3b3;"
    }

    html = estimator_html_repr(pipeline)

    for old, new in replacements.items():
        html = html.replace(old, new)
    display(HTML(html))
`
![Screenshot 2023-05-14 100143](https://github.com/scikit-learn/scikit-learn/assets/95470962/7d12d2ad-3db7-4947-9fd5-41c588c8cbfe)


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: solo-driven
issue_26359: state: open
title: Selecting Lasso via cross-validation data leakage
body: ### Describe the issue linked to the documentation

```py
model = make_pipeline(StandardScaler(), LassoCV(cv=20)).fit(X, y)
```

Optimizing Alpha for lasso involves data leakage because standard scaling is applied on X and y in one go, instant for each fold.

It is used in the example Selecting Lasso via cross-validation
https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_model_selection.html




### Suggest a potential alternative/fix

_No response_
user: nielsbijl
issue_26353: state: open
title: ENH Allow to extract and reconstruct patches using a specific stride
body: ### Added stride parameter to `extract_patches_2d` and `reconstruct_from_patches_2d` functions.

In sklearn.feature_extraction.image, current `extract_patches_2d` function extract patches with stride forced to 1. This function use another function `_extract_patches` that can manage different strides with the parameter `extraction_step`.
This change adds the `stride` parameter in `extract_patches_2d` and `reconstruct_from_patches_2d`.

### Notes
#### 1. The way overlapping parts of patches are managed has changed
Executing this code:
```python
import numpy as np
from sklearn.feature_extraction.image import extract_patches_2d, reconstruct_from_patches_2d
for img_size, patch_size, stride in [(128, (5, 5), 1), (1500, (64, 64), 32)]:
    img = np.random.rand(img_size, img_size)
    print(f"{img_size}x{img_size} image, {patch_size[0]}x{patch_size[1]} patches, stride {stride}")
    p = extract_patches_2d(img, patch_size, stride=stride)
    print("Reconstruction from p:", p.shape)
    %time r = reconstruct_from_patches_2d(p, img.shape, stride=stride)
    print()
```
##### With the current sklearn implementation
```
128x128 image, 5x5 patches, stride 1
Reconstruction from p: (15376, 5, 5)
CPU times: user 35.2 ms, sys: 3.6 ms, total: 38.8 ms
Wall time: 38.6 ms

1500x1500 image, 64x64 patches, stride 32
Reconstruction from p: (2025, 64, 64)
CPU times: user 1.22 s, sys: 7.86 ms, total: 1.23 s
Wall time: 1.23 s
```
##### With the implementation from this pull request
```
128x128 image, 5x5 patches, stride 1
Reconstruction from p: (15376, 5, 5)
CPU times: user 64.6 ms, sys: 1.07 ms, total: 65.7 ms
Wall time: 65.5 ms

1500x1500 image, 64x64 patches, stride 32
Reconstruction from p: (2025, 64, 64)
CPU times: user 24.7 ms, sys: 16.5 ms, total: 41.3 ms
Wall time: 41.3 ms
```

With stride=1, the new implementation is 2x slower.
With stride=32, the new implementation is 30x faster.

#### 2. Reconstruction is perfect depending on stride value
However, we could `np.pad` image in `extract_patches_2d` so that all parts in image are extracted, and remove the padding in `reconstruct_from_patches_2d` given `image_size`.

user: REDVM
issue_26343: state: open
title: Warning due to `lscpu` on MacOS on nightly builds
body: I created a conda environment containing the `scipy-dev` packages (to solve the deprecation warning shown in our CI). Once I built scikit-learn, I get the following error when importing:

```python
> ipython
Python 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:12:31) [Clang 14.0.6 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.13.2 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sklearn
/bin/sh: lscpu: command not found

In [2]: sklearn.__version__
Out[2]: '1.3.dev0'
```

@ogrisel @jeremiedbb Do you know if it is coming from `threadpoolctl` or `joblib`? Basically the `command not found` is raised because I am on MacOS
user: glemaitre
issue_26335: state: open
title: ENH `check_classification_targets` raises a warning when unique classes > 50% of `n_samples`
body: #### Reference Issues/PRs
Towards #16399

#### What does this implement/fix? Explain your changes.
Addresses the first point of https://github.com/scikit-learn/scikit-learn/issues/16399#issue-560932649

#### Any other comments?
user: rprkh
labels: Waiting for Second Reviewer: First reviewer is done, need a second one!
issue_26330: state: open
title: ENH Add `sample_weight` parameter to `OneHotEncoder`'s `.fit`
body: Fixes #26248.

This PR adds `sample_weight` parameter to `OneHotEncoder.fit` function and calculate level frequencies using `sample_weight` sum instead of counting. Also, this enables the ranking of categories accordingly for the use of `max_categories`.
user: vitaliset
issue_26329: state: open
title: API Allow users to pass instances of `DistanceMetric` directly to `metric` keyword arguments
body: # Motivation

SIMD intrinsics can accelerate pairwise distance computation by a factors of ~2.5-3.5x for `float64` data, and ~5-6x for `float32` data (benchmarked by this gist: https://gist.github.com/Micky774/bd1b8394fdaa82b25dcdfc111835c19b).
<details> 
<summary>
Plots
</summary>

![01ef290e-2055-47c8-ae0f-3fa009c478ef](https://user-images.githubusercontent.com/34613774/236366585-29759ac2-e31c-49e9-a95b-ead4ddc05e69.png)
![373fb964-81cf-4550-9875-39e0d5de28bd](https://user-images.githubusercontent.com/34613774/236366588-98f068a9-54e9-408f-b0c7-58533b9f44a0.png)

</details>

These benefits translate effectively into computation-bound estimators, such as `KNeighborsRegressor` (based on https://github.com/scikit-learn/scikit-learn/pull/26267):

<details> 
<summary>
Plots
</summary>

![1d7664fd-870d-4a7a-8f1a-7f25a0029d9c](https://user-images.githubusercontent.com/34613774/236367317-9b05d175-22d8-4deb-be82-9d49c0241b7e.png)

![f9aa8ab6-2dd3-4b47-9e20-0a6ea4e17cc3](https://user-images.githubusercontent.com/34613774/236367325-d412c4ee-5283-41d7-b898-05632b4c3269.png)

</details>

# Alternatives Considered

As discussed in https://github.com/scikit-learn/scikit-learn/issues/26010 and https://github.com/Micky774/scikit-learn/pull/11, while there is a significant preference towards avoiding implementing SIMD-based solutions within scikit-learn at this time. I do believe that there is a reasonable way to maintain such work (at least up to `SSE3` instructions), however a better-accepted solution is to create a plug-in for `DistanceMetric` and offer the SIMD-accelerated implementations as an engine. While this is indeed a good solution in the long run, there is still much work needed to be done on the plug-in API (https://github.com/scikit-learn/scikit-learn/issues/22438). Working on a separate engine/plug-in for `DistanceMetric` while the API is still being solidified and https://github.com/scikit-learn/scikit-learn/pull/25535 is still unmerged is probably going to do more harm than good by adding one more moving part to the mix and slowing down the review process.

# Suggested Solution

Allow users to pass instances of `DistanceMetric` directly to `metric` keyword arguments. This is backwards compatible and doesn't require any significant new infrastructure (mainly small changes to validation and updated docs/tests). This enables third-party libraries to provide their own accelerated solutions immediately.

In practice, this involves changes mainly in the following: 

- [x] [`ArgKmin`](https://github.com/scikit-learn/scikit-learn/pull/26267)
- [ ] `RadiusNeighbors`
- [ ] `ArgKminClassMode`
- [ ] `pairwise_distances`
- [ ] `pairwise_distances_argmin`

This will allow us to enable the functionality in parts of the following estimators (non-exhaustive):
```
- NearestNeighbors
- KNeighborsRegressor
- KNeighborsClassifier
- RadiusNeighborsRegressor
- RadiusNeighborsClassifier
- DBSCAN
- OPTICS
- Isomap
- TSNE (self.method != "exact")
- KernelDensity
- AffinityPropagation
- Birch
- MeanShift
- NearestCentroid
```

Notes:

1. `pairwise_distanecs` can't actually use the `DistanceMetric` in its current state, _however_ once https://github.com/scikit-learn/scikit-learn/pull/25561 is completed, it can benefit from accelerated `DistanceMetric` options as well.
2. Currently `{KD, Ball}Tree` support passing `DistanceMetric` through the `metric` argument, however do not support `DistanceMetric32` (see: https://github.com/scikit-learn/scikit-learn/pull/25914)

# Implementation

I have a sample implementation of this for `KNeighborsRegressor`, which is achieved by enabling this functionality for `ArgKmin` along with updating parameter validation in `NeighborsBase`; please see https://github.com/scikit-learn/scikit-learn/pull/26267. 
user: Micky774
issue_26326: state: open
title: Add a facility that allows random forest classifiers to be combined after training
body: ### Describe the workflow you want to enable

In a federated environment, I have federation elements that build private random forest classifiers, which I would like to combine after the fact into a single random forest.  



### Describe your proposed solution

See the "alternatives" section.  

### Describe alternatives you've considered, if relevant

Stacking might suffice as a work-around, although I'd like to avoid that.  

As a throw-away experiment, simply concatenating all the constituent decision tree estimators into a common estimators_ array (and adjusting the count) seems to work superficially, but clearly isn't good practice.   

In addition, this approach can fail, if, say, we try to combine random forest instance #1 which has classes_ of [dog, cat] and forest #2 which has classes_ of [cow, dog, cat].    To address that concern, I looked at forcing the union of all possible classes (over all the forests) into the resultant combined forest, and the underlying trees.   This appears to work at some level, but doesn't handle misshapen oob_decision_function_ which is shaped according n_classes_.    

Another approach to dealing with classes_ heterogeneity is to make sure  each federation forest is exposed to the full gamut of potential classes during training.  (Even then, one worries about the order of the elements found in classes_ : [dog,cat] vs [cat,dog]).    It appears that classes_ is constructed before any bootstrap sampling, so, assuming we can rely on that implementation detail, and we expose each federation to a consistently ordered and specially constructed "gamut" pre-pended to their X, we can (hopefully) expect all forest instances to have identical classes_ with the same elements in the same order.  That, in turn, would allow easier combining of the forests.     Ensuring complete exposure via the "gamut" might also impact accuracy.    (The training "gamut" is a minimal set of X records that produce all possible y categorical values). 
 

### Additional context

R provides a combine() operator.  


user: davedice
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_26321: state: open
title: Duality gap computation in covariance.GraphicalLasso yields negative values.
body: ### Describe the bug

The computation of the duality gap in `_dual_gap(emp_cov, precision_, alpha)` of `GraphicalLasso` uses the definition from `Duchi et al., 2012`. 
However, their duality gap is expressed given a _feasible_ dual variable. In the implementation, it is applied to a primal variable that doesn't necessarily satisfy the dual variable's feasibility constraints.
This results in potentially negative values for the duality gap.

The primal problem reads

$$P(\Theta) = \min_\Theta \varphi(\Theta) + \nu(\Theta)$$
with
$$\varphi(\Theta) =  -\log \det (\Theta)  + \langle S, \Theta \rangle \text{ and } \nu(\Theta) =  \Vert \Lambda \odot \Theta \Vert_1$$
The Fenchel-Rockafellar dual problem is then given by

$$D(W) = \min_{W}  \varphi^{\star} (W) + \nu^{\star} (-W)$$
$$= \min_{W} -\log \det(S - W) - p \quad \text{s.t. } \quad \vert W_{ij} \vert < \lambda_{ij}$$
We can then compute the duality gap as follows:
$$G(\Theta, W) = P(\Theta) + D(W)$$
Only for a *feasible* $W$ can we actually take $\theta = (S - W)^{-1}$ and thus simplify the duality gap expression and fall back on the implemented formula:
$$G(\Theta) = \langle S, \Theta \rangle + \Vert \Lambda \odot \Theta \Vert_1 - p$$ 
@mathurinm 

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.covariance import GraphicalLasso

np.random.seed(0)
X = np.random.randn(100, 50)
emp_cov = X.T @ X / len(X)

clf = GraphicalLasso(verbose=True, tol=1e-20).fit(emp_cov)
```

### Expected Results

The expected results would be positive values for the duality gap at all iterations.

### Actual Results

Here are the duality gaps for the first 4 iterations : 
```
[graphical_lasso] Iteration   0, cost  5.24e+01, dual gap 6.745e-01
[graphical_lasso] Iteration   1, cost  5.24e+01, dual gap -3.808e-05
[graphical_lasso] Iteration   2, cost  5.24e+01, dual gap -4.379e-09
[graphical_lasso] Iteration   3, cost  5.24e+01, dual gap 5.049e-09

```

### Versions

```shell
System:
    python: 3.10.6 (main, Oct 24 2022, 16:07:47) [GCC 11.2.0]
executable: /home/cpouliquen/anaconda3/envs/these-can/bin/python3.10
   machine: Linux-5.4.0-148-generic-x86_64-with-glibc2.27

Python dependencies:
          pip: 22.2.2
   setuptools: 64.0.2
      sklearn: 1.0.2
        numpy: 1.22.3
        scipy: 1.8.1
       Cython: 0.29.33
       pandas: 1.5.1
   matplotlib: 3.5.3
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```
```

user: Perceptronium
issue_26311: state: open
title: ENH add post fit calibration to HGBT
body: #### Reference Issues/PRs
Closes #22435

#### What does this implement/fix? Explain your changes.
This PR adds the parameter `post_fit_calibration` to `HistGradientBoostingRegressor`. In the case of a non-canonical link, i.e. only for `loss="gamma"`, after the fit is more or less finished, a constant is added to the raw_predictions in link space such that on the training data (minus the `validation_fraction`), the balance property is fulfilled: the weighted average of predictions (`predict`) equals weighted average of observations, i.e. `np.average(y, weights=sample_weight)`.

#### Any other comments?
TODO list:
- [x] add functionality
- [x] add tests

user: lorentzenchr
issue_26308: state: open
title: Use scipy.stats.yeojohnson PowerTransformer
body: Inside `PowerTransformer`, we should use [`scipy.stats.yeojohnson`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.yeojohnson.html#scipy-stats-yeojohnson) instead of our own implementation.

`scipy.stats.yeojohnson` was release with scipy 1.2.0. With PR #24665, we now have even 1.5.0 as minimum.

Edit: Note that https://github.com/scipy/scipy/issues/18389 to be release in scipy 1.12 will also resolve #23319.
user: lorentzenchr
labels: Moderate: Anything that requires some knowledge of conventions and best practices
Refactor: Code refactor
issue_26307: state: open
title: KNeighborsClassifier OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata
body: ### Describe the bug

when i run flask app on 56 core cpus,it show this warning and app exit, when i change \site-packages\joblib\externals\loky\backend\context.py can solved

os_cpu_count = min(os.cpu_count() or 1,12)

cpu_count_user = min(_cpu_count_user(os_cpu_count),12)

### Steps/Code to Reproduce

#knn is KNeighborsClassifier 
knn= joblib.load('knn.model')
knn.predict(data)

### Expected Results

KNeighborsClassifier.predict no bug on 56 cpus or more cpus

### Actual Results

OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata; flask app exit,no Error show.

### Versions

```shell
>>> import sklearn
>>> sklearn.show_versions()

System:
    python: 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
executable: E:\Program Files\Python310\python.exe
   machine: Windows-10-10.0.14393-SP0

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.1
   setuptools: 65.5.0
        numpy: 1.24.3
        scipy: 1.10.1
       Cython: None
       pandas: 1.4.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: E:\Program Files\Python310\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 112

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: E:\Program Files\Python310\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
        version: 0.3.21
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 24

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: E:\Program Files\Python310\Lib\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 24
```

user: linuxl7
labels: Needs Investigation: Issue requires investigation
issue_26296: state: open
title: Add support for `dtype` arg in `MultiLabelBinarizer`
body: ### Describe the workflow you want to enable

Similar to other transformers (e.g., `OneHotEncoder`), support an arg `dtype` in `MultiLabelBinarizer`.

### Describe your proposed solution

Add an arg `dtype` in `MultiLabelBinarizer`.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: bryant1410
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_26292: state: open
title: Add support for bools in `SimpleImputer`
body: ### Describe the workflow you want to enable

Suppose you wanna impute a bool array. Because it has NaNs, it's gonna be of dtype float and work fine:

```pycon
>>> np.asarray([[True, False, np.nan]]).dtype
dtype('float64')
```

However, now suppose that the value you pass actually has no NaNs (e.g., because the current data happens to have no NaNs), so there's nothing to impute. In this case, the array is of type bool:

```pycon
>>> np.asarray([[True, False]]).dtype
dtype('bool')
```

Which makes the imputation fail because `bool` isn't supported:

```python
import numpy as np
from sklearn.impute import SimpleImputer

SimpleImputer(strategy="most_frequent").fit(np.asarray([[True, False]]))
```

This previous code generates an exception:

```
Traceback (most recent call last):
  # ...
  File ".../lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 140, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File ".../lib/python3.10/site-packages/sklearn/base.py", line 878, in fit_transform
    return self.fit(X, **fit_params).transform(X)
  File ".../lib/python3.10/site-packages/sklearn/impute/_base.py", line 390, in fit
    X = self._validate_input(X, in_fit=True)
  File ".../lib/python3.10/site-packages/sklearn/impute/_base.py", line 352, in _validate_input
    raise ValueError(
ValueError: SimpleImputer does not support data with dtype bool. Please provide either a numeric array (with a floating point or integer dtype) or categorical data represented either as an array with integer dtype or an array of string values with an object type.
```

This is because `np.asarray([[True, False]]).dtype.kind` is "b", which is not in {"i", "u", "f", "O"} and so it fails.

### Describe your proposed solution

My solution is to support `bool`. I believe it shouldn't be a great effort.

### Describe alternatives you've considered, if relevant

An alternative solution is for me to create a custom imputer class that copy-pastes the behavior I want from `SimpleImputer` and makes it work for bools.

### Additional context

_No response_
user: bryant1410
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_26284: state: open
title: ENH Add per feature max_categories for OrdinalEncoder
body: #### Reference Issues/PRs
Closes #26013.


#### What does this implement/fix? Explain your changes.

This PR allows a user to specify per-feature max categories for an `OrdinalEncoder` by passing `max_categories` as a dictionary mapping a valid feature name to its corresponding maximum number of output categories. Since identifying infrequent categories is already done per feature, instead of applying the global limit when `max_categories` is an integer, the current feature name is retrieved and its corresponding value in the dictionary is used as the upper limit.


#### Any other comments?

Currently, this PR assumes that for any `X` that is not a pandas DataFrame, the feature names are the ones generated by `get_feature_names_out`. Any improvements or suggestions would be appreciated.
user: Andrew-Wang-IB45
issue_26277: state: open
title: Support `max_bins > 255` in Hist-GBDT estimators and categorical features with high cardinality
body: As originally sketched in https://github.com/scikit-learn/scikit-learn/pull/26268#issuecomment-1520504489 there might be a way to enable support for arbitrary high values of `max_bins` for both categorical and numerical features. This may not be super critical for numerical features, but this would **enable categorical features of arbitrary cardinality**, which is desirable.

The rough idea is to internally map an input categorical feature into **multiple **binned** features** (probably `num_categories // 255 + 1` features) and to update the `Splitter` and the predictors to treat that group of features as a single feature.
user: NicolasHug
issue_26268: state: open
title: ENH Support categories with cardinality higher than max_bins in HistGradientBoosting
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Related to https://github.com/scikit-learn/scikit-learn/pull/24907


#### What does this implement/fix? Explain your changes.
This PR enables support for categorical features that have cardinality greater than `max_bins` and categories that are encoded above `max_bins`. This is enabled with a new `on_high_cardinality_categories="bin_infrequent"` parameter.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
issue_26266: state: open
title: TST Interaction between `class_weight` and `sample_weight`
body: **Reference Issues/PRs**

Towards #21504.

**What does this implement/fix? Explain your changes.**

This pull request introduces two test functions for scikit-learn classifier that check the interaction between `class_weight` and `sample_weight` parameters as proposed by @jjerphan:

- `check_interaction_of_class_and_sample_weight_excluding_class`: Verifies that setting a class weight to zero is equivalent to excluding samples of that class from calibration, even with non-uniform sample weights.

- `check_interaction_of_class_and_sample_weight_excluding_samples`: Confirms that setting sample weights to zero is equivalent to removing corresponding samples from training, even with non-uniform class weights.

**Any other comments?**

- Classifiers that failed for a reason other than `assert_allclose_dense_sparse`:.

    - `NuSVC` failed the `check_sample_weights_invariance` test (present check) due to an error in the value of the nu parameter. This error persisted here.

    - `SVC` is giving an error due to "dual coefficients or intercepts are not finite".

    - `LinearSVC` is failing in the `assert_allclose_dense_sparse`, but not for the reasons one would expect. The output of `decision_function` method is giving an array of shape `(n_samples,)`, instead of `(n_samples, n_classes)` when trained with zero sample weight for one of the classes (and `n_classes=3`). I suppose this is a bug. (Worthy of an issue?)

<details>
  <summary>Minimal code with the same error of SVC</summary>

```python
from sklearn import __version__
print(__version__)
>>> 1.2.0
from sklearn.svm import SVC
from sklearn.datasets import make_classification

n_classes = 3
X, y = make_classification(
    n_samples=200,
    n_classes=n_classes,
    n_informative=2 * n_classes,
    random_state=0,
)
class_weight_dict = {cls: 1 if cls != 0 else 0 for cls in range(n_classes)}
print(class_weight_dict)
>>> {0: 0, 1: 1, 2: 1}

svc = SVC(class_weight=class_weight_dict).fit(X, y)
>>> ValueError: The dual coefficients or intercepts are not finite. The input data may contain large values and need to bepreprocessed.
```
</details>


<details>
  <summary>Minimal code with the same error of LinearSVC</summary>

```python
from sklearn import __version__
print(__version__)
>>> 1.2.0
import numpy as np
from sklearn.datasets import make_classification
from sklearn.svm import LinearSVC

n_classes = 3
X, y = make_classification(
    n_samples=200,
    n_classes=n_classes,
    n_informative=2 * n_classes,
    random_state=0,
)

sample_weight = np.ones_like(y)
sample_weight_zero_weight_first = np.where(y == 0, 0, sample_weight)

svc_sw = (
    LinearSVC(random_state=0)
    .fit(X, y, sample_weight=sample_weight_zero_weight_first)
)
>>> ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.

print(svc_sw.decision_function(X).shape)
>>> (200,)

svc = (
    LinearSVC(random_state=0)
    .fit(X, y, sample_weight=sample_weight)
)
>>> ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.

print(svc.decision_function(X).shape)
>>> (200, 3)
```
</details>


user: vitaliset
issue_26253: state: open
title: FIX Raises an erorr in vectorizers when output is pandas
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Related to #26210

#### What does this implement/fix? Explain your changes.
This PR has the Vectorizers raise an error when the global configuration sets the output to pandas.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
issue_26248: state: open
title: Add sample_weight parameter to OneHotEncoder.fit(...)
body: ### Describe the workflow you want to enable

I have a dataset with huge number of duplicates - so in order to speed-up learning process I prefer to remove these duplicates and pass `sample_weight=duplicate_counts` to the `fit(..)` method of estimator. I override the `fit(..)` method of estimator and remove duplicates in this wrapper.

But If I use `OneHotEncoder` as pre-processing step, it produces sparse output - so the input to `estimator.fit(..)` is sparse. My dataset is large, so I strongly want to use sparse output of `OneHotEncoder`.

And effective duplicate removal in sparse input is not a trivial task - it must use sorting or hashing.  So I want to perform duplicate removal and `sample_weight` calculation before one hot encoding.

But I use the `min_frequency` parameter of `OneHotEncoder` - and want level frequencies to be calculated with the use of `sample_weight`. 

### Describe your proposed solution

Add `sample_weight` parameter to `OneHotEncoder.fit(..)` function and calculate level frequencies using `sample_weight` parameter.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: ArseniuML
issue_26222: state: open
title: RFC Memory usage of tests
body: I recently tried to run the scikit-learn test suite with the [pytest-memray](https://github.com/bloomberg/pytest-memray) plugin. Here are the tests that allocate the most memory.

Here is are the top 10 worst offenders on my local machine (with CPython 3.11):

```
pytest --memray --most-allocations=10
```

<details>

```
==================================================================== MEMRAY REPORT =====================================================================
Allocations results for sklearn/tests/test_random_projection.py::test_random_projection_numerical_consistency[GaussianRandomProjection]

         📦 Total memory allocated: 348.7MiB
         📏 Total allocations: 208
         📊 Histogram of allocation sizes: |▃ ▁ ▁ ▂ █|
         🥇 Biggest allocating functions:
                - within_tol:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/numeric.py:2361 -> 134.2MiB
                - assert_array_compare:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/testing/_private/utils.py:799 -> 94.7MiB
                - _gaussian_random_matrix:/Users/ogrisel/code/scikit-learn/sklearn/random_projection.py:196 -> 63.1MiB
                - fit:/Users/ogrisel/code/scikit-learn/sklearn/random_projection.py:417 -> 31.6MiB
                - isclose:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/numeric.py:2378 -> 7.9MiB


Allocations results for sklearn/ensemble/tests/test_iforest.py::test_iforest_with_n_jobs_does_not_segfault

         📦 Total memory allocated: 324.9MiB
         📏 Total allocations: 1052
         📊 Histogram of allocation sizes: |▆       █|
         🥇 Biggest allocating functions:
                - __init__:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/sparse/_coo.py:190 -> 129.7MiB
                - _array_indexing:/Users/ogrisel/code/scikit-learn/sklearn/utils/__init__.py:186 -> 65.5MiB
                - __init__:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/sparse/_coo.py:191 -> 64.8MiB
                - _check:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/sparse/_coo.py:279 -> 32.4MiB
                - _check:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/sparse/_coo.py:278 -> 32.4MiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch[0-3-5000-0.1]

         📦 Total memory allocated: 150.0MiB
         📏 Total allocations: 3118
         📊 Histogram of allocation sizes: |▂ ▅ █ ▅ █|
         🥇 Biggest allocating functions:
                - c2c:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/fft/_pocketfft/basic.py:31 -> 68.7MiB
                - transform:/Users/ogrisel/code/scikit-learn/sklearn/kernel_approximation.py:217 -> 34.3MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - safe_sparse_dot:/Users/ogrisel/code/scikit-learn/sklearn/utils/extmath.py:193 -> 703.1KiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch[2.5-3-5000-2.5]

         📦 Total memory allocated: 149.9MiB
         📏 Total allocations: 3136
         📊 Histogram of allocation sizes: |▂ ▅ ▂ ▅ █|
         🥇 Biggest allocating functions:
                - c2c:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/fft/_pocketfft/basic.py:31 -> 68.7MiB
                - transform:/Users/ogrisel/code/scikit-learn/sklearn/kernel_approximation.py:217 -> 34.3MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - safe_sparse_dot:/Users/ogrisel/code/scikit-learn/sklearn/utils/extmath.py:193 -> 703.1KiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch[2.5-3-5000-1]

         📦 Total memory allocated: 149.9MiB
         📏 Total allocations: 3136
         📊 Histogram of allocation sizes: |▂ ▅ ▂ ▅ █|
         🥇 Biggest allocating functions:
                - c2c:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/fft/_pocketfft/basic.py:31 -> 68.7MiB
                - transform:/Users/ogrisel/code/scikit-learn/sklearn/kernel_approximation.py:217 -> 34.3MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - safe_sparse_dot:/Users/ogrisel/code/scikit-learn/sklearn/utils/extmath.py:193 -> 703.1KiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch[2.5-3-5000-0.1]

         📦 Total memory allocated: 149.9MiB
         📏 Total allocations: 3136
         📊 Histogram of allocation sizes: |▂ ▅ ▂ ▅ █|
         🥇 Biggest allocating functions:
                - c2c:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/fft/_pocketfft/basic.py:31 -> 68.7MiB
                - transform:/Users/ogrisel/code/scikit-learn/sklearn/kernel_approximation.py:217 -> 34.3MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - safe_sparse_dot:/Users/ogrisel/code/scikit-learn/sklearn/utils/extmath.py:193 -> 703.1KiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch[0-3-5000-1]

         📦 Total memory allocated: 149.9MiB
         📏 Total allocations: 3108
         📊 Histogram of allocation sizes: |▂ ▅ ▂ ▅ █|
         🥇 Biggest allocating functions:
                - c2c:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/fft/_pocketfft/basic.py:31 -> 68.7MiB
                - transform:/Users/ogrisel/code/scikit-learn/sklearn/kernel_approximation.py:217 -> 34.3MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - safe_sparse_dot:/Users/ogrisel/code/scikit-learn/sklearn/utils/extmath.py:193 -> 703.1KiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch[0-3-5000-2.5]

         📦 Total memory allocated: 149.9MiB
         📏 Total allocations: 3108
         📊 Histogram of allocation sizes: |▂ ▅ ▂ ▅ █|
         🥇 Biggest allocating functions:
                - c2c:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/fft/_pocketfft/basic.py:31 -> 68.7MiB
                - transform:/Users/ogrisel/code/scikit-learn/sklearn/kernel_approximation.py:217 -> 34.3MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - safe_sparse_dot:/Users/ogrisel/code/scikit-learn/sklearn/utils/extmath.py:193 -> 703.1KiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_skewed_chi2_sampler

         📦 Total memory allocated: 103.2MiB
         📏 Total allocations: 217
         📊 Histogram of allocation sizes: |▄ █    |
         🥇 Biggest allocating functions:
                - test_skewed_chi2_sampler:/Users/ogrisel/code/scikit-learn/sklearn/tests/test_kernel_approximation.py:179 -> 103.0MiB
                - test_skewed_chi2_sampler:/Users/ogrisel/code/scikit-learn/sklearn/tests/test_kernel_approximation.py:174 -> 117.2KiB
                - test_skewed_chi2_sampler:/Users/ogrisel/code/scikit-learn/sklearn/tests/test_kernel_approximation.py:173 -> 117.2KiB
                - wrapper:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/pytest_memray/plugin.py:157 -> 2.1KiB


Allocations results for sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_poisson

         📦 Total memory allocated: 99.4MiB
         📏 Total allocations: 327946
         📊 Histogram of allocation sizes: |█ ▅ ▄ ▂  |
         🥇 Biggest allocating functions:
                - split_next:/Users/ogrisel/code/scikit-learn/sklearn/ensemble/_hist_gradient_boosting/grower.py:583 -> 1.5MiB
                - split_next:/Users/ogrisel/code/scikit-learn/sklearn/ensemble/_hist_gradient_boosting/grower.py:587 -> 1006.7KiB
                - _array_indexing:/Users/ogrisel/code/scikit-learn/sklearn/utils/__init__.py:186 -> 472.7KiB
                - dot:<__array_function__ internals>:200 -> 468.8KiB
```

</details>


It seems to confirm that high memory usage in some tests can cause the PyPy CI to crash: #25117.

Questions:

- should we define an official memory limit for individual tests (e.g. 50 MB or less) and then work towards rewriting the offending tests to work within that budget?
- alernatively shall we mark some tests (or test parametrizations) explicitly as high memory tests and use a flag to skip those tests on sensitive CI configs such as PyPy?
- should we use `pytest-memray` on some CI runs? It seems to be compatible with our use of `pytest-xdist` and the overhead seems low enough in quick experiments (to be confirmed on our CI in a PR).

user: ogrisel
issue_26211: state: open
title: Possible inconsistency between documentation and code for user guide of PLSCanonical
body: ### Describe the issue linked to the documentation

Recently I've been learning Partial Least Squares (PLS), and learned that there are some variations of PLS, namely PLS-Canonical, PLS-SVD, PLS2 and PLS1 (mainly from the [User Guide of cross decomposition](https://scikit-learn.org/stable/modules/cross_decomposition.html#cross-decompositionhttps://) in the python's scikit-learn library. I could understand most of it, and have read the reference that page has provided [A survey of Partial Least Squares (PLS) methods, with emphasis on the two-block case, by JA Wegelin](https://stat.uw.edu/sites/default/files/files/reports/2000/tr371.pdf). Nevertheless, I still have questions about the details of predicting $Y$ based on new samples.

In the user guide, it states that to compute $Y$, we need 

> The idea is to try to predict the transformed targets $\Omega$ as a function of the transformed samples $\Xi$, by computing $\alpha \in \mathbb{R}$ such that $\Omega = \alpha \Xi$.
>
> ...
>
> ... **and as a result the coefficient matrix $\beta = \alpha P \Delta^T$**.

However in code, `self._coef_` is computed by
```python
self._coef_ = np.dot(self.x_rotations_, self.y_loadings_.T)
self._coef_ = (self._coef_ * self._y_std).T
```
This code is located at [L355](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/cross_decomposition/_pls.py#L355) of `_pls.py`.

If I am correct, here `self.x_rotations_` is a transformation matrix such that `X @ self.x_rotations_` would be $\Xi$. Therefore `self.x_rotations_` would be $P$. And `self.y_loadings_` is $\Delta$. **As a result, the code is computing $\beta = P \Delta^T $, which I think is inconsistent with the documentation described above.**


### Suggest a potential alternative/fix

In my honest opinion, maybe we should add the computation of $\alpha$ into the code. 

Moreover, I think $\alpha$ shouldn't be a scalar; instead, $\alpha$ should be a vector of length $R$, where $R$ is the number of columns of $\Xi$, i.e. the number of latent scores. Each $\xi_i$ should be multiplied by $\alpha_i$ before multiplying $\Delta^T$:

$$ 
\beta = P 
\left[ \begin{matrix}
 \alpha_1 & & \\
& \ddots & \\
& & \alpha_R 
 \end{matrix}  \right]
 \Delta^T 
$$
user: rk-terence
labels: Needs Investigation: Issue requires investigation
issue_26208: state: open
title: Pass optional column names to parallel tree function in using the self.feature_names_in attribute.
body: #### Reference Issues/PRs
Example: Fixes #26140 

#### What does this implement/fix? Explain your changes.
In case of using a pandas DataFrame as the X variable type, every tree inside the forest will set its `feature_names_in_` attribute equal the list of the columns of set pandas DataFrame 

#### Any other comments?


user: wesselhuising
issue_26204: state: open
title: Fix PLSR n_components documentation.
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes incorrect restriction on n_components in PLSRegression docstring.

#### What does this implement/fix? Explain your changes.
The PLSRegression class docstring incorrectly states that n_components is upper-bounded by n_targets, but this is only the case for PLSCanonical. This fix removes that restriction from the docstring, so that it now aligns with the description for PLSRegression in the User Guide.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: lhdgsn
issue_26202: state: open
title: [mrg] ENH Add custom_range argument for partial dependence - version 2
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Fixes https://github.com/scikit-learn/scikit-learn/issues/20890 and is a followup to the stale PR - https://github.com/scikit-learn/scikit-learn/pull/21033 

#### What does this implement/fix? Explain your changes.

This PR allows users to specify a custom_range of values to calculate partial depedency for some or all of the features.

The api is custom_range={feature: array-like of grid values}.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: stephenpardy
issue_26192: state: open
title: Add sampling uncertainty on precision-recall and ROC curves
body: #### Reference Issues/PRs
Closes #25856 


#### What does this implement/fix? Explain your changes.

Add sampling uncertainty on precision-recall and ROC curves.
See more details in the Issue above.
user: stephanecollot
issue_26189: state: open
title: ENH reuse parent histogram in HGBT
body: #### Reference Issues/PRs
None

#### What does this implement/fix? Explain your changes.
This PR reuses the parent's histogram for the feature that was split on. This saves a little time.

#### Any other comments?
The implementation is no beauty. If we want to include this improvement, suggestions or alternative PRs are welcome.
user: lorentzenchr
issue_26179: state: open
title: SLEP006: default routing
body: In the context of: 
- https://github.com/scikit-learn/scikit-learn/issues/25776
- https://github.com/scikit-learn/scikit-learn/issues/26050
 
 we've also discussed the possibility of developing a default routing strategy for certain metadata. In most cases this is `sample_weight` and probably `groups` in scikit-learn itself.

This would mean, after the introduction of SLEP006, this code would work, and route `sample_weight` to every object which accepts it (since that's what we think is suitable in this case):

```py
# The exact API is TBD
sklearn.set_config(enable_auto_routing=True)

GridSearchCV(
	LogisticRegression(), 
	scorer=a_scorer_supporting_sample_weight, ...
).fit(..., sample_weight=sw)
```

We can then decide whether or not we want the auto-routing to be enabled by default. One major thing to consider here is that with auto-routing enabled, behavior of the same code can change from version to version for two main reasons:

- we change our mind / find bugs / etc in the routing, and how we want to route things
- estimator A might not support `sample_weight` in version `x`, but starts supporting it in version `x+1`, and with default routing the behavior of the same code changes

## Notes
- auto-routing can always be overridden by the user, for more advanced usecases.
- third party developers can use the same mechanism, for `sample_weight` or other metadata if they see fit
- we might deem https://github.com/scikit-learn/scikit-learn/issues/26050 unnecessary if we develop this feature

cc @scikit-learn/core-devs 
user: adrinjalali
issue_26176: state: open
title: fix(areaScores): consistency b/w aupr auroc
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes #24381


#### What does this implement/fix? Explain your changes.

- replace `ValueError` with `Userwarning`
- update -0.0 with 0.0 

user: a-parida12
issue_26170: state: open
title: RFC BayesSearchCV in scikit-learn
body: ### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26141

<div type='discussions-op-text'>

<sup>Originally posted by **earlev4** April 11, 2023</sup>
**Hi! First off, thanks so much to the excellent work done by all the scikit-learn contributors! The project is truly a gift and your work is greatly appreciated!**

I still consider myself a novice when it comes to scikit-learn. In my usage, I typically will attempt to use GridSearchCV when searching for the best parameters. However, depending on the search space, GridSearchCV might not always be the best option and can be computationally expensive and time-consuming in some scenarios. RandomizedSearchCV can be an alternative in these situations, but does not always seem to provide the best parameters compared to scikit-optimize BayesSearchCV. In my humble opinion, scikit-optimize BayesSearchCV seems to be a nice compromise between GridSearchCV and RandomizedSearchCV, providing good parameters in a reasonable time.

Unfortunately, scikit-optimize BayesSearchCV seems to be no longer supported. The last commit was in 2021. As of NumPy 1.24, NumPy now results in an error, unless a workaround of `np.int = int` is used. This is just one example. There are numerous issues that have not been touched since 2021. It would be a shame to lose a project such as scikit-optimize BayesSearchCV and _humbly ask the contributors of scikit-learn if a similar version could be implemented in scikit-learn_.

**Thanks so much for your consideration! Looking forward to the discussion.**</div>

----------------------------------------

I thought `scikit-optimize` was a maintained project, but it seems it isn't, and I agree it'd be nice for the community to have a maintained `BayesSearchCV` available. 

I'm not sure if we should include it here, or `scikit-learn-extra`, but would be nice to have it.
user: adrinjalali
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_26164: state: open
title: LinearRegression with zero sample_weights is not the same as excluding those rows
body: ### Describe the bug

Excluding rows having `sample_weight == 0` in `LinearRegression` does not give the same results.

### Steps/Code to Reproduce

```python
from collections import Counter
import numpy as np
from sklearn.linear_model import LinearRegression

results = []
for i in range(100):
    rng = np.random.RandomState(i)
    n_samples, n_features = 10, 5
    X = rng.rand(n_samples, n_features)
    y = rng.rand(n_samples)
    reg = LinearRegression()
    sample_weight = rng.uniform(low=0.01, high=2, size=X.shape[0])
    sample_weight_0 = sample_weight.copy()
    sample_weight_0[-5:] = 0
    y[-5:] *= 1000  # to make excluding those samples important
    reg.fit(X, y, sample_weight=sample_weight_0)
    coef_0, intercept_0 = reg.coef_.copy(), reg.intercept_

    reg.fit(X[:-5], y[:-5], sample_weight=sample_weight[:-5])
    print(f"{coef_0=}")
    print(f"{reg.coef_=}")
    results.append(np.allclose(reg.coef_, coef_0, rtol=1e-6))
print(Counter(results))
```

### Expected Results

Always `True`.

### Actual Results

```python
Counter({True: 79, False: 21})  # it fails 20% of the time
```

The print statement gives:
```
coef_0 =    array([ 1.43516166, -1.78826443,  0.15365526,  1.82233166, -1.6       ])
reg.coef_ = array([ 2.24022351, -1.04917851,  0.45341088,  0.80315086, -0.25726798])
```


### Versions

```shell
System:
    python: 3.9.15 (main, Nov 15 2022, 05:24:15)  [Clang 14.0.0 (clang-1400.0.29.202)]
   machine: macOS-13.3.1-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.2.2
        numpy: 1.23.5
        scipy: 1.10.1
       Cython: 0.29.33

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
```
```

user: lorentzenchr
issue_26148: state: open
title: Cloned estimators have identical randomness but different RNG instances
body: ### Describe the bug

Cloned estimators have identical randomness but different RNG instances. According to documentation, it should be the other way around: different randomness but identical RNG instances.

Related #25395 

The User Guide [says](https://scikit-learn.org/stable/common_pitfalls.html#controlling-randomness):
> For an optimal robustness of cross-validation (CV) results, pass RandomState instances when creating estimators

> ```python
> rf_inst = RandomForestClassifier(random_state=np.random.RandomState(0))
> cross_val_score(rf_inst, X, y)
> ```
> ...
> Since `rf_inst` was passed a `RandomState` instance, each call to `fit` starts from a different RNG. As a result, the random subset of features will be different for each folds

In regards to cloning, the same reference says:
> ```python
> rng = np.random.RandomState(0)
> a = RandomForestClassifier(random_state=rng)
> b = clone(a)
> ```
> Moreover, `a` and `b` will influence each-other since they share the same internal RNG: calling `a.fit` will consume `b`’s RNG, and calling `b.fit` will consume `a`’s RNG, since they are the same. 

The actual behaviour does not follow this description. 

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn import clone
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_validate
from sklearn.ensemble import RandomForestClassifier

rng = np.random.RandomState(0)
X, y = make_classification(random_state=rng)
rf = RandomForestClassifier(random_state=rng)

d = cross_validate(rf, X, y, return_estimator=True, cv=2)
rngs = [e.random_state for e in d['estimator']]
# estimators corresponding to different CV runs have different but identical RNGs:
print(rngs[0] is rngs[1]) # False
print(all(rngs[0].randint(10, size=10) == rngs[1].randint(10, size=10))) # True

rf_clone = clone(rf)
rngs = [rf.random_state, rf_clone.random_state]
print(rngs[0] is rngs[1]) # False
print(all(rngs[0].randint(10, size=10) == rngs[1].randint(10, size=10))) # True
```

### Expected Results

True False True False

### Actual Results

False True False True

### Versions

```shell
Tested on a two-week-old dev build and also on the following version (Kaggle)

System:
    python: 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:53)  [GCC 9.4.0]
executable: /opt/conda/bin/python3.7
   machine: Linux-5.15.90+-x86_64-with-debian-bullseye-sid

Python dependencies:
          pip: 22.3.1
   setuptools: 59.8.0
      sklearn: 1.0.2
        numpy: 1.21.6
        scipy: 1.7.3
       Cython: 0.29.34
       pandas: 1.3.5
   matplotlib: 3.5.3
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

user: avm19
labels: Needs Decision: Requires decision
issue_26140: state: open
title: RandomForest not passing feature names to trees and creating warnings.
body: ### Describe the bug

I fit a decision forest with training data that includes feature names. When I call predict_proba on the forest everything is fine. When I call rf.estimators_[0].predict_proba it will warn that it was not trained with feature names.


### Steps/Code to Reproduce
```python
import numpy as np
from sklearn.ensemble import RandomForestClassifier

print(sklearn.__version__)

data = np.random.normal([1,2,3,4,5,6,7,8,9,10], size=(1000, 10))
feature_names = [f'F{i}' for i in range(10)]
df = pd.DataFrame(data=data, columns=feature_names)
y = np.ones(1000)
y[500:] = 0

rf = RandomForestClassifier()
rf.fit(df, y)

print(f"Feature names in Forest: {rf.feature_names_in_} ")

print('Classing pred proba on forest')
rf.predict_proba(df)
print('Done calling pred proba on forest')

# THIS GIVES A WARNING
rf.estimators_[0].predict_proba(df)
```

### Expected Results

No warnings

### Actual Results
```
Feature names in Forest: ['F0' 'F1' 'F2' 'F3' 'F4' 'F5' 'F6' 'F7' 'F8' 'F9'] 
Classing pred proba on forest
Done calling pred proba on forest
.....lib/python3.8/site-packages/sklearn/base.py:432: UserWarning: X has feature names, but DecisionTreeClassifier was fitted without feature names
  warnings.warn(
```
### Versions

```shell
sk>>> sklearn.show_versions()

System:
    python: 3.8.16 (default, Mar  2 2023, 03:21:46)  [GCC 11.2.0]
executable: ....../bin/python
   machine: Linux-5.15.0-60-generic-x86_64-with-glibc2.17

Python dependencies:
      sklearn: 1.2.2
          pip: 23.0.1
   setuptools: 65.6.3
        numpy: 1.23.1
        scipy: 1.9.0
       Cython: None
       pandas: 1.5.3
   matplotlib: None
       joblib: 1.1.1
threadpoolctl: 2.2.0
```

user: howarth
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_26128: state: open
title: HistGradientBoosting counts and sample weights
body: Related issues: #25210

### Current State
`HistGradientBootingClassifier` and `HistGradientBootingRegressor` both:
- Calculate the sample size `count` in histograms
- Use `count` for splitting (mostly excluding split candidates)
- Save the `count` in the final trees and use it in partial dependence computations.

### Proposition
1. Evaluate if removing `count` from the histograms (LightGBM only sums gradient and hessian in histograms, no count) gives a good speed-up.
  Edit: LightGBM uses an approximate count based on the hessian to check for min sample size. So this might not be what we want.
3. Add an option to save counts and sample weights to final trees at the very end of `fit` (where the binned training `X` is still available).
4. Use partial dependence `method='recursion'` if the above option was set, else use `method='brute'`.

Why?
#25431 concluded that adding weights to the trees is too expensive. The above proposition gives a user a clear choice: Faster training time or faster pdp afterwards.

user: lorentzenchr
labels: Needs Decision: Requires decision
Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
Needs Decision - Backward Compatibility: Requires decision regarding backward compatibility
issue_26120: state: open
title: FEA add TunedThresholdClassifier meta-estimator to post-tune the cut-off threshold
body: superseded #16525
closes #16525 
closes https://github.com/scikit-learn/scikit-learn/issues/8614
closes https://github.com/scikit-learn/scikit-learn/pull/10117
supersedes https://github.com/scikit-learn/scikit-learn/pull/10117

build upon https://github.com/scikit-learn/scikit-learn/pull/26037

relates to #4813

### Summary

We introduce a `TunedThresholdClassifier` that intends to post-tune the cut-off points to convert a soft decision of the `decision_function` or `predict_proba` to a hard decision provided by `predict`.

### Important features to have in mind:

**`objective_metric`**: the objective metric is set to either a metric to be maximized or a pair of metrics, one to be optimized under the constraint of the other (to find a trade-off). Additionally, we can pass a cost/gain-matrix that could be used to optimize a business metric. For this case, we are limited to constant cost/gain. In the future, we can think of cost/gain that depends on the matrix `X` but we would need to be able to forward meta-data to the scorer (a good additional use case for SLEP006 @adrinjalali).

**`cv` and `refit`**: we provide some flexibility to pass refitted model and single train/test split. We add limitations and documentation to caveats with an example.

### Point to discuss

- Are we fine with the name `TunedThresholdClassifier`? Shall instead have something about "threshold" (e.g. `ThresholdTuner`)?
- We are using the term `objective_metric`, `constraint_value` and `objective_score`. Is the naming fine? An alternative to "objective" might be "utility"?

### Further work

I implemented currently a single example that shows the feature in the context of post-tuning of the decision threshold.

The current example is using a single train/test split for the figure and I think it would be nice to have some ROC/precision-recall curve obtained from cross-validation to be complete. However, we need some new features to be implemented.

I also am planning to analyse the usage of this feature on the problem of calibration on imbalanced classification problems. The feeling on this topic is that resampling strategies involved an implicit tuning of the decision threshold at the cost of a badly calibrated model. It might be better to learn a model on the imbalanced problem directly, making sure that it is well calibrated and then post-tune the decision threshold for "hard" prediction. In this case, you get the best of two worlds: a calibrated model if the output of `predict_proba` is important to you and an optimum hard predictor for your specific utility metric. However, this is going to need some investigation and will be better suited for another PR.
user: glemaitre
issue_26109: state: open
title: HalvingRandomSearchCV and HalvingGridSearchCV do not support multimetric scoring.
body: According to https://scikit-learn.org/stable/modules/grid_search.html, 

_HalvingRandomSearchCV and HalvingGridSearchCV do not support multimetric scoring._

When will this be implemented?

When trying to use multimeric scoring today, I get: 

`ValueError: scoring parameter must be a string, a callable or None. Multimetric scoring is not supported.`
user: gynther-k
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_26094: state: open
title: FIX GroupShuffleSplit raises a ValueError for NaN
body: #### Reference Issues/PRs
Fixes #24486

#### What does this implement/fix? Explain your changes.
A ValueError is raised if the input to GroupShuffleSplit contains a `pd.NA` or `np.nan`:

```
import sklearn
import pandas as pd
import numpy as np
from sklearn.model_selection import GroupShuffleSplit

data = pd.DataFrame({"clusters": [1, 2, 3, pd.NA, np.nan],
                    "x" : [0,1,2,3,4]})

splitter = GroupShuffleSplit(test_size=.2, n_splits=2, random_state = 7)
split = splitter.split(data, groups=data['clusters'])
train_inds, test_inds = next(split)
```

Output:
```
ValueError: Input groups contain NaN.
```

#### Any other comments?
The errors for the failing tests seem to be due to [pandas not being found](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=53933&view=logs&j=689a1c8f-ff4e-5689-1a1a-6fa551ae9eba&t=0b7e60d2-0e3c-59af-8129-1150b3e7bf0c&l=211910) on some builds.



user: rprkh
issue_26090: state: open
title: Rename quantile parameter in QuantileRegressor and HistGradientBoostingRegressor
body: [`QuantileRegressor`](https://scikit-learn.org/dev/modules/generated/sklearn.linear_model.QuantileRegressor.html#sklearn.linear_model.QuantileRegressor) and [`HistGradientBoostingRegressor`](https://scikit-learn.org/dev/modules/generated/sklearn.ensemble.HistGradientBoostingRegressor.html#sklearn.ensemble.HistGradientBoostingRegressor) both have a `quantile` parameter. But the meaning is the **probability level** of the quantile to be estimated/predicted, not the quantile itself.

R's [`quantile`](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/quantile.html) function calls this parameter `probs` for probability.

Can we consider to rename it? Possible suggestions:
- `quantile_level`
- `probability`
user: lorentzenchr
labels: Needs Decision - API: Requires decision regarding API
issue_26083: state: open
title: RFC/API (Array API) mixing devices and data types with estimators
body: Right now, if the user fits an estimator using a `pandas.DataFrame`, but passes a `numpy.ndarray` during `predict`, they get a warning due to missing feature names.

The situation is only to get more complicated as we're adding support for more types via array API.

Some related issues here are:
- device: data during `fit` sits on a GPU, but a CPU is used for predict (with the same data type)
- types: using one type to `fit`, and use another type during `predict`: how do we handle this both in terms of device and the type? Do we let the operator figure out if they can coerce the data into the type which can be used?
- persistence: how do we let users fit on one device, but load on another device
- estimator conversion: do we let users convert an estimator which is fit using one type/device, to an estimator compatible with another type/device?

I vaguely remember us talking about some of these issues, but I don't see any active discussion. I might have missed something.

Related: in a library like `pytorch`, you can decide which device is going to be used when you load a model's weights.

cc @thomasjpfan 
user: adrinjalali
issue_26063: state: open
title: Use `np.uint8` as default dtype for `OneHotEncoder` instead of `np.float64`
body: ### Describe the workflow you want to enable

`sklearn.preprocessing.OneHotEncoder` should use as `dtype` the `np.uint8` by default instead of `np.float64` as it is currently. I don't see the reasoning behind using `np.float64`, this if anything only causes memory explosions and potentially (?) even slows down the computation.

### Describe your proposed solution

Make `np.uint8` as the default `dtype` parameter of `sklearn.preprocessing.OneHotEncoder`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: smith558
issue_26062: state: open
title: Add `class_weight` feature to KNeighbors, GradientBoosting and AdaBoost classifiers
body: ### Describe the workflow you want to enable

I propose adding a `class_weight` parameter to the [KNeighborsClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html), [GradientBoostingClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) and [AdaBoostClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html) in Scikit-learn to handle class imbalances in datasets. Many real-world datasets suffer from class imbalance, which can lead to biased and suboptimal models. By including a `class_weight` parameter in these classifiers, users can easily adjust the importance of each class during training, allowing for better model performance in imbalanced scenarios.

### Describe your proposed solution

For each of the classifiers, I propose adding a `class_weight` parameter with the following options:

- `None`: No class weighting is applied (default behavior).
- `'balanced'`: Class weights are automatically computed based on the number of samples in each class.
- `{class_label: weight}`: A dictionary providing custom weights for each class.

The `class_weight` parameter should be used during the training phase to modify the loss function for the classifiers, accounting for the class weights provided. This modification will ensure that the model takes into consideration the class imbalance while training.

For MLPClassifier, this feature has been requested in issue #9113. Implementing this for the other classifiers will follow a similar pattern.

### Describe alternatives you've considered, if relevant

An alternative approach is to use resampling techniques to balance the dataset prior to training. However, this method can have some drawbacks, such as potential information loss in the case of under-sampling, or increased training time and overfitting risk in the case of over-sampling. Adding a `class_weight` parameter directly to the classifiers offers a more efficient and flexible way to handle class imbalance, without the need for preprocessing the data.

Another alternative is to use third-party libraries that offer implementations of these classifiers with built-in support for class weights. However, this may not be ideal for users who prefer to work exclusively with Scikit-learn or have constraints that prevent the use of additional libraries.

### Additional context

Some classifiers in Scikit-learn, such as the [RandomForestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) and [SVC](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html), already support the `class_weight` parameter. Adding this functionality to the proposed classifiers would make the library more consistent and user-friendly.

This feature is also already proposed for the MLPClassifier, in:
- Issue: #9113
- PR: #25326

### Tracking

- [ ] KNeighborsClassifier
- [ ] GradientBoostingClassifier
- [ ] AdaBoostClassifier 
user: EwoutH
issue_26061: state: open
title: Store spectral embeddings in SpectralClustering
body: ### Describe the workflow you want to enable

Save the spectral embeddings used for clustering in the [SpectralClustering](https://github.com/scikit-learn/scikit-learn/blob/9aaed4987/sklearn/cluster/_spectral.py#L394) class and make them accessible through an attribute, e.g. `maps_`, to make easier post-processing on the clusters.

### Describe your proposed solution

Optionally return the maps in the [spectral_clustering](https://github.com/scikit-learn/scikit-learn/blob/9aaed4987/sklearn/cluster/_spectral.py#L194) method with a new parameter:

```
def spectral_clustering(
    affinity,
    *,
    n_clusters=8,
    n_components=None,
    eigen_solver=None,
    random_state=None,
    n_init=10,
    eigen_tol="auto",
    assign_labels="kmeans",
    verbose=False,
    return_maps=False
):
    
    ...
    
    if return_maps:
        return maps, labels
    else:
        return labels
```

Store `maps_` attribute in the `fit` method of the `SpectralClustering` class:

```
self.maps_, self.labels_ = spectral_clustering(
            self.affinity_matrix_,
            n_clusters=self.n_clusters,
            n_components=self.n_components,
            eigen_solver=self.eigen_solver,
            random_state=random_state,
            n_init=self.n_init,
            eigen_tol=self.eigen_tol,
            assign_labels=self.assign_labels,
            verbose=self.verbose,
            return_maps=True
        )
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: matteo-bastico
issue_26050: state: open
title: SLEP006: globally setting request values
body: One of the issues raised in https://github.com/scikit-learn/scikit-learn/issues/25776 and https://github.com/scikit-learn/scikit-learn/issues/23928 is about a code such as following to work under SLEP6:

```py
est = AdaBoostClassifier(LogisticRegression())
est.fit(X, y, sample_weight=sw)
```
or
```py
GridSearchCV(
	LogisticRegression(), 
	scorer=a_scorer_supporting_sample_weight, ...
).fit(..., sample_weight=sw)
```
without having to change the code, or w/o having to write too much boilerplate code, or w/o having to explicitly set request values for `sample_weight` for these _usual_ usecases.

In order to allow the above pattern, the proposal is to allow users to set request values globally. These values can either be set as a configuration, or via dedicated methods. The user code at the end would look like:

```py
sklearn.set_fit_request(sample_weight=True)
sklearn.set_score_request(sample_weight=True)

GridSearchCV(
	LogisticRegression(), 
	scorer=a_scorer_supporting_sample_weight, ...
).fit(..., sample_weight=sw)
```

What the above code does, is to set he default value to `REQUESTED` instead of `ERROR_IF_PASSED` for `sample_weight`, on `fit` and `score`methods / scorers everywhere.

## API

A generic way to achieve this would be to set global default request values, on any method, for any metadata, which can generalize to `groups` and other metadata present in third party libraries.

Alternatively, we could only allow this for `sample_weight` (I rather not do this), and do it with something like:

```py
sklearn.set_config(fit_requests_sample_weight=True)
```
or if we don't want to distinguish between methods:
```py
sklearn.set_config(sample_weight_request=True)
```

My proposal would be:

```py
sklearn.set_{method}_request(metadata=value)
```
which follows the API on the estimator level, except that here we're setting it on the library scope.

## Context Manager

An addition to the above API, we can provide a context manager for the same purpose. If we expose this through a config, it's automatically done via `config_context`. If we follow another API, we can also provide the corresponding context managers.

## Effect on SLEP6's examples

Here we look at how the [examples in the SLEP](https://scikit-learn-enhancement-proposals.readthedocs.io/en/latest/slep006/proposal.html) would change.

### Nested Grouped Cross Validation

```py
weighted_acc = make_scorer(accuracy_score).set_score_request(sample_weight=True)
log_reg = LogisticRegressionCV(cv=GroupKFold(), scoring=weighted_acc).set_fit_request(
    sample_weight=True
)
cv_results = cross_validate(
    log_reg,
    X,
    y,
    cv=GroupKFold(),
    metadata={"sample_weight": my_weights, "groups": my_groups},
    scoring=weighted_acc,
)
```
becomes:

```py
sklearn.set_fit_request(sample_weight=True)
sklearn.set_score_request(sample_weight=True)

weighted_acc = make_scorer(accuracy_score)
log_reg = LogisticRegressionCV(cv=GroupKFold(), scoring=weighted_acc)
cv_results = cross_validate(
    log_reg,
    X,
    y,
    cv=GroupKFold(),
    metadata={"sample_weight": my_weights, "groups": my_groups},
    scoring=weighted_acc,
)
```

And users can still override the default global value at the estimator level, for example, the next example:

```py
weighted_acc = make_scorer(accuracy_score).set_score_request(sample_weight=True)
log_reg = LogisticRegressionCV(cv=group_cv, scoring=weighted_acc).fit_request(
    sample_weight=False
)
cross_validate(
    log_reg,
    X,
    y,
    cv=GroupKFold(),
    metadata={"sample_weight": weights, "groups": groups},
    scoring=weighted_acc,
)
```

becomes

```py
sklearn.set_fit_request(sample_weight=True)
sklearn.set_score_request(sample_weight=True)

weighted_acc = make_scorer(accuracy_score)
log_reg = LogisticRegressionCV(cv=group_cv, scoring=weighted_acc).fit_request(
    sample_weight=False
)
cross_validate(
    log_reg,
    X,
    y,
    cv=GroupKFold(),
    metadata={"sample_weight": weights, "groups": groups},
    scoring=weighted_acc,
)
```

### Unweighted Feature selection
Here there's no request value for `SelectKBest` set since it doesn't support `sample_weight`:

```py
weighted_acc = make_scorer(accuracy_score).set_score_request(sample_weight=True)
log_reg = LogisticRegressionCV(cv=GroupKFold(), scoring=weighted_acc).set_fit_request(
    sample_weight=True
)
sel = SelectKBest(k=2)
pipe = make_pipeline(sel, log_reg)
pipe.fit(X, y, sample_weight=weights, groups=groups)
```

it becomes:

```py
sklearn.set_fit_request(sample_weight=True)
sklearn.set_score_request(sample_weight=True)

weighted_acc = make_scorer(accuracy_score)
log_reg = LogisticRegressionCV(cv=GroupKFold(), scoring=weighted_acc)
sel = SelectKBest(k=2)
pipe = make_pipeline(sel, log_reg)
pipe.fit(X, y, sample_weight=weights, groups=groups)
```

### More advanced use cases with aliased routing

Aliased routing is not supported at the moment, and if the user wants to pass different sample weights to different objects, they'll need to use the verbose API.

## Open Questions
- How should the API exactly look like? Do we like the proposal here?
- Do we want to also implement the context manager?

------------------------------------------

## `sample_weight="auto"`

We also talked about good default request values for `sample_weight` specifically (https://github.com/scikit-learn/scikit-learn/issues/25776#issuecomment-1492877533), to allow users to follow our recommendations. This would be enabled by doing something like:

```py
sklearn.set_config(auto_sample_weight_routing=True)
```
or more explicitly
```py
sklearn.set_{method}_request(sample_weight='auto')
```

This would enable recommended routing, whatever we decide that to be. This is NOT a blocker for releasing SLEP6 and can be implemented after it's released, and recommendations are subject to change, and we tell users that they can change.

There shall be a separate issue to talk about the default values and how the end user API should look like. It is only mentioned here to give context on what we plan to do.

cc @scikit-learn/core-devs 
user: adrinjalali
issue_26045: state: open
title: SLEP006: introduction of metadata routing through a feature flag
body: In https://github.com/scikit-learn/scikit-learn/issues/25776 we started talking about introducing SLEP6 through a feature flag. This would mean, the user can control whether SLEP6 is enabled or not. This will be done via our global configs:

```py
import sklearn
sklearn.set_config(slep6="enabled")
# or
sklearn.set_config(enable_slep6=True)
# or ...
```

This also allows users to use this in a context manager, via `config_context`, if they wish to.

## Default value

We can decide on the default value and the path forward. Probably the easiest option here would be for the default value of the setting to be off/`False`, which means at first the introduction of SLEP6 wouldn't break anything and users' code runs as it would before.


### Transition

The following steps are required to completely switch to SLEP6:

1. Land the implementation on `main`, and have it released in version `a`
2. In version `b`, start warning users who pass any metadata, that they should be switching to SLEP6 and that the default will change in version `c` to be enabled by default, and that old routing would be removed in version `d`
	- This can be done by raising a warning everywhere we do old routing and we observe a metadata/fit param not to be `None` (and set the warning to be raised once)
3. Remove old routing in version `d`
4. Deprecate `sklearn.set_config(enable_slep6=True)` in version `e`, and remove in version `f`, since once the old routing is removed from the code base, doesn't make sense to keep the flag in the configs.

Constraints and open questions:
- `a <= b`: but they can also be the same, i.e. we start raising the warning as soon as the feature is shipped.
- `b < c`: and our usual deprecation cycle is two releases, i.e. `c = b + 2`, but we can allow a longer deprecation cycle here if we want.
- `c <= d`: we could have `c` and `d` to happen at the same time, but we probably would want to give some buffer in between.
- `d <= e < f`: since setting the flag's value doesn't do anything and setting it to `False` is not allowed once we remove the old routing, we should probably have `e == d`, but we can remove it with a gracious deprecation period.

My proposal here:
- `b == a`: warn immediately to start the move cycle)
- `c == b + 3`: give 1.5 years for users to move to the new syntax
- `d == c + 1`: 6 months period here
- `d == e`: deprecate the flag immediate since users can't really use it at this point anyway
- `f == e + 4`: let the config not raise for 2 years

## Implementation

The implementation follows the same pattern as mentioned in https://github.com/scikit-learn/scikit-learn/issues/25776#issuecomment-1490777659:

```python
class MyMetaEstimator(...):

    def fit(self, X, y, **fit_params):
        if SLEP006_ON:
            params = process_routing(self, "fit", fit_params)
        else:
            # define params using backward compatible semantics
            params = ...

        self.estimator_ = clone(self.estimator).fit(X, y, **params.estimator.fit)
```

with the added complexity that:
- Introducing SLEP6 is also adding metadata to other methods such as `transform`, etc, which means these would be accepted and added to the signature of metaestimators such as `Pipeline`. If the user passes anything to those newly introduce parameters w/o enabling the feature flag, we `raise`.
- Introducing SLEP6 also deprecates certain parameters of functions such as `cross_val_score`, e.g. `fit_prams -> metadata/params`. This deprecation message and the switch will depend on how we decide on the transition as explained in previous section.

We shall also implement this for the PRs for metaestimators that are already open, and apply this to the ones we've already merged into the feature branch (`sample-props`).

## Third party developers

They can decide if they want to follow our way of introducing the new syntax, or to introduce it with a breaking change. For them this introduces an added complexity. They can ideally tie things to how we change settings in our config.

-------------------------

This came as a result of the meeting we held on 31.03.2022: https://github.com/scikit-learn/scikit-learn/issues/25776#issuecomment-1492176862

cc @scikit-learn/core-devs 
user: adrinjalali
issue_26040: state: open
title:  Implementation of an additional patch extraction strategy. 
body: ### Describe the workflow you want to enable

I think it would be valuable to add a function that extracts non-overlapping patches from an image. This alternative patch extraction strategy might be more suitable for certain use cases where overlapping patches are not desired or could lead to redundant information. I propose adding the following function to the codebase:



With this new function, you could extract non-overlapping patches from an image. This would be a valuable addition to the existing code, as it provides an alternative patch extraction strategy that might be more suitable for certain use cases.

### Describe your proposed solution

Here's an example of how you could implement this new function:
def extract_non_overlapping_patches_2d(image, patch_size):
    """Extract non-overlapping patches from a 2D image.
    
    Parameters
    ----------
    image : ndarray of shape (image_height, image_width) or \
        (image_height, image_width, n_channels)
        The original image data. For color images, the last dimension specifies
        the channel: a RGB image would have `n_channels=3`.
    patch_size : tuple of int (patch_height, patch_width)
        The dimensions of one patch.

    Returns
    -------
    patches : array of shape (n_patches, patch_height, patch_width) or \
        (n_patches, patch_height, patch_width, n_channels)
        The collection of non-overlapping patches extracted from the image.
    """
    i_h, i_w = image.shape[:2]
    p_h, p_w = patch_size

    if p_h > i_h or p_w > i_w:
        raise ValueError("Patch dimensions should be smaller than the image dimensions.")

    n_rows = i_h // p_h
    n_cols = i_w // p_w
    n_patches = n_rows * n_cols

    image = image[:n_rows * p_h, :n_cols * p_w]  # Crop the image to fit the non-overlapping patches
    image = image.reshape((i_h, i_w, -1))
    n_colors = image.shape[-1]

    patches = np.empty((n_patches, p_h, p_w, n_colors))

    patch_idx = 0
    for row in range(n_rows):
        for col in range(n_cols):
            patch = image[row * p_h:(row + 1) * p_h, col * p_w:(col + 1) * p_w]
            patches[patch_idx] = patch
            patch_idx += 1

    if patches.shape[-1] == 1:
        return patches.reshape((n_patches, p_h, p_w))
    else:
        return patches

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: Aryan-Mishra24
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_26038: state: open
title: Update documentation with new non-overlapping patch extraction function
body: 
Title: Implementation of Non-overlapping Patch Extraction Function in Scikit-Learn's feature_extraction.image Module

Comments:

The extract_non_overlapping_patches_2d function extracts non-overlapping patches from a 2D image, which can be a useful alternative to the existing extract_patches_2d function.
The function takes in an image and patch size as input parameters, and returns a collection of non-overlapping patches extracted from the image.
The function first checks if the patch dimensions are smaller than the image dimensions, and then calculates the number of non-overlapping patches that can be extracted from the image.
The image is then cropped to fit the non-overlapping patches, and the patches are extracted and stored in an array.
If the image is grayscale, the function returns an array of shape (n_patches, patch_height, patch_width). If the image is color, the function returns an array of shape (n_patches, patch_height, patch_width, n_channels).
user: Aryan-Mishra24
issue_26032: state: open
title: [ENH, DRAFT] Implement Precomputed distances in the Pairwise Distances Reduction framework
body: #### Reference Issues/PRs
Towards: #https://github.com/scikit-learn/scikit-learn/issues/25888

This is just a draft and I will mark it ready for review when I am done.

#### What does this implement/fix? Explain your changes.
- Adds `PrecomputedDistanceMatrix` subclass of `DatasetsPair`, which has the precomputed distance matrix between two pairs of arrays
- Adds the relevant API and changes in the logic to support a user passing in `'precomputed'` as the metric.

#### Any other comments?
Thanks @jjerphan for pointing me in the right direction initially with his commit at https://github.com/jjerphan/scikit-learn/commit/78b3aa41767d910de116b18e6e4bbce9a931a02b#diff-2c0fbe478f4ed2459849f7227487a1df0f7cac915508a970dd96220418ca5784
user: adam2392
issue_26026: state: open
title: `Fmax` score (or maximum of F1/Fbeta)
body: ### Describe the workflow you want to enable

The maximum of F1 across thresholds is a well-studied metric and it is both robust and valid in binary and multilabel classification problems.  Basically, it can be computed with `precision_recall_curve`, as shown below, for binary problems.  For multilabel problems, I'm not sure if there is an efficient way to do this without looping thru all labels.

### Describe your proposed solution

For binary:
```
def get_fmax(preds: np.ndarray, ys: np.ndarray, beta = 1.0, pos_label = 1):
    """
    Radivojac, P. et al. (2013). A Large-Scale Evaluation of Computational Protein Function Prediction. Nature Methods, 10(3), 221-227.
    """
    precision, recall, thresholds = precision_recall_curve(y_true = ys, probas_pred = preds, pos_label = pos_label)
    numerator = (1 + beta**2) * (precision * recall)
    denominator = ((beta**2 * precision) + recall)
    fbeta = np.divide(numerator, denominator, out=np.zeros_like(numerator), where=(denominator!=0))
    
    return np.nanmax(fbeta), thresholds[np.argmax(fbeta)]
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: jasperhyp
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_26024: state: open
title: Make more of the "tools" of scikit-learn Array API compatible
body: 🚨 🚧 This issue requires a bit of patience and experience to contribute to 🚧 🚨 

Please mention this issue when you create a PR, but please don't write "closes #26024" or "fixes #26024".

scikit-learn contains lots of useful tools, in addition to the many estimators it has. For example [metrics](https://scikit-learn.org/stable/modules/classes.html#sklearn-metrics-metrics), [pipelines](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.pipeline), [pre-processing](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing) and [mode selection](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection). These are useful to and used by people who do not necessarily use an estimator from scikit-learn. This is great.

The fact that many users install scikit-learn "just" to use `train_test_split` is a testament to how useful it is to provide easy to use tools that do the right(!) thing. Instead of everyone implementing them from scratch because it is "easy" and making mistakes along the way.

In this issue I'd like to collect and track work related to making it easier to use all these "tools" from scikit-learn even if you are not using Numpy arrays for your data. In particular thanks to the Array API standard it should be "not too much work" to make things usable with data that is in an array that conforms to the Array API standard.

There is work in #25956 and #22554 which adds the basic infrastructure needed to use "array API arrays". Right now you need to checkout #25956 (this is part of the reason why this is a draft issue).

The goal of this issue is to make code like the following work:
```python
>>> from sklearn.preprocessing import MinMaxScaler
>>> from sklearn import config_context
>>> from sklearn.datasets import make_classification
>>> import torch
>>> X_np, y_np = make_classification(random_state=0)
>>> X_torch = torch.asarray(X_np, device="cuda", dtype=torch.float32)
>>> y_torch = torch.asarray(y_np, device="cuda", dtype=torch.float32)

>>> with config_context(array_api_dispatch=True):
...     # For example using MinMaxScaler on PyTorch tensors
...     scale = MinMaxScaler()
...     X_trans = scale.fit_transform(X_torch, y_torch)
```

The first step is to compile a list of tools that are in scope for this. The next step (or maybe part of the first) is to check which of them already "just work". After that is done we can start the work (one PR per class/function) making changes. Hopefully by then #25956 is ready or already merged.

---

* The main reason I created this issue is to get some feedback, let people know I'm thinking about this and to have a place to work on this/collect notes. If you want to join in that would be great, but for now there is no need for code changes/PRs.
user: betatim
labels: Meta-issue: General issue associated to an identified list of tasks
issue_26013: state: open
title: Add per feature "maximum category" counts to `OrdinalEncoder`
body: ### Describe the workflow you want to enable

This is a follow up task for #25677

It would be nice to allow users to [specify a per feature number of maxcategories](https://github.com/scikit-learn/scikit-learn/pull/25677#discussion_r1132500192) instead of having a global limit as implemented in #25677.

More details in the linked comment.

### Describe your proposed solution

Allow users to pass a list of shape `(n_features,)` or a dict mapping column name to `max_categories` values to specify the number of maximum categories per feature.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: betatim
issue_26012: state: open
title: ENH add verbose option for gpr
body: #### Reference Issues/PRs

Fixes #25740

#### What does this implement/fix? Explain your changes.
In sklearn/gaussian_process/_gpr.py, added a verbose parameter. The default value is 0. If verbose >= 1, define a list  self.explored_theta to track all tested theta in the regression process, and define another list self.explored_theta_log to track the corresponding log_marginal_likelihood. If the user is passing self-defined optimizer and let verbose >= 1, then this self-defined optimizer should return corresponding array of explored theta and log marginal likelihood itself.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: ROMEEZHOU
issue_26010: state: open
title: Introduce SIMD intrinsics for `_dist_metrics.pyx`
body: # Context
Pairwise distance computation is an essential part of many estimators in scikit-learn, and can take up a significant portion of run time in certain workflows. I believe that we may achieve significant performance gains in several (perhaps most) distance metric implementations by leveraging SIMD intrinsics.

# Proof of Concept

I built a quick proof of concept just to see what kinds of performance gains we could observe with a potentially-naive implementation of SIMD intrinsics. I chose to optimize the `ManhattanDistance.dist` function. This implementation uses intrinsics found in `SSE{1,2,3}`. To ensure that the instructions are supported, it checks for the presence of the `SSE3`  instruction set (`SSE3` implies `SSE{1,2}`) and provides the optimized implementation if so. Otherwise it provides a dummy implementation just to appease Cython, and the main function falls back to the current implementation on `main`. Note that on most modern hardware, support for `SSE3` is a reasonable expectation (indeed numpy assumes it is always present when optimization is enabled). For the specific implementation referred to here, please take a look at this PR: https://github.com/Micky774/scikit-learn/pull/11

Note that the full benefit of the intrinsics are gained when compiling with `-march="native"`, however the benefit is still significant when compiling with `-march="nocona"`, as is often default (e.g when following the scikit-learn development instructions on linux).

# Benchmarks
The following benchmarks were produced by this gist: https://gist.github.com/Micky774/567a5fa199c05d90c4c08625b077840e

### **Summary: The SIMD implementations are ~2x faster than the current implementation for `float32` and 1.5x faster for `float64`.**

<details>
<summary>Plots</summary>

![f2b1f1e8-59b0-4ec5-b91c-fe1d19abd9ec](https://user-images.githubusercontent.com/34613774/228374474-99ee13b3-228d-4c53-9dcc-3a8f8b639a9f.png)
</details>

# Discussion
I haven't looked too deeply into this yet, as first I wanted to see whether there was interest in the venture. I would love to hear what the other maintainers' thoughts are regarding exploring this route in a bit more detail. Obviously SIMD implementations will bring with them added complexity, but the performance gains are pretty compelling. In my opinion, the tradeoff is worth it.

CC: @scikit-learn/core-devs
user: Micky774
labels: Needs Decision: Requires decision
issue_26008: state: open
title: #23112: showcase idea to cache last step
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs

Fixes #23112 
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.

Demo discussion state of #23112

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: windiana42
issue_25978: state: open
title: Typo in the range of n_components in PLSRegression
body: There is a typo in the range of n_components in PLSRegression. https://scikit-learn.org/stable/modules/generated/sklearn.cross_decomposition.PLSRegression.html#sklearn.cross_decomposition.PLSRegression

Should be in "[1, min(n_samples - 1, n_features)]", not in "[1, min(n_samples, n_features, n_targets)]".

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

![image](https://user-images.githubusercontent.com/3051351/228107725-bfb4d66c-8429-4986-98fb-5b566c3a4382.png)


#### What does this implement/fix? Explain your changes.


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: li1127217ye
issue_25977: state: open
title: [MRG] DOC GPs: log_marginal_likelihood() log(theta) input
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #24786

#### What does this implement/fix? Explain your changes.

Document the fact that the input argument `theta` to GaussianProcess{Regressor,Classifier}'s `log_marginal_likelihood()` method is actually expected to be `log(theta)`, as this is used in the optimizer code path when `self.kernel_.theta` is passed and where the latter triggers a getter that returns `log(theta)`. This is now consistent with other doc strings where "log-transformed theta" is mentioned.

In the method doc strings, only add "log-transformed". In the GPR user guide add an extra note.

Add two tests, one for GPR and GPC.

#### Any other comments?

No changelog needed. This change is small and only corrects documentation and adds two tests.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: elcorto
issue_25970: state: open
title: Allow NearestNeighbor radius_neighbors to accept an array for the `radius` keyword argument
body: ### Describe the workflow you want to enable

In KDTree and BallTree, there is a way to pass in a radius array that queries nearest-neighbors for each sample point for different radii.

- BallTree: https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.BallTree.html#sklearn.neighbors.BallTree.query_radius
- KDTree: https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KDTree.html#sklearn.neighbors.KDTree.query_radius

However, in `NearestNeighbors` class, the https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestNeighbors.html#sklearn.neighbors.NearestNeighbors.radius_neighbors function only allows a single radius. 

In my current application, I am first computing the distance to the say Kth nearest-neighbor, `R` for each sample point in some space, and then for each sample point, computing the number of neighbors within that radius in some other spaces:

```
neigh = NearestNeighbors(
        n_neighbors=k,
        algorithm=algorithm,
        metric=metric,
        n_jobs=n_jobs
    ).fit(dists)

dists, _ = neigh.kneighbors()
radius_per_sample = dists[:, -1]

# compute on the subspace of XZ
 xz_data = np.hstack((X, Z))
 neigh = compute_nn(
        xz_data, algorithm=algorithm, metric='forest', k=knn_here, n_jobs=n_jobs)

# here, though I have to loop over each sample
for idx in range(n_samples):
        num_nn = neigh.radius_neighbors(radius=radius_per_sample[idx])

# If I repeat this multiple times across different subspaces, it is a bit cumbersome
```

### Describe your proposed solution

Allow `radius` parameter in `NearestNeighbors.radius_neighbors` to be an array of shape `(n_samples,)`, which would have a similar functionality as the KDTree and BallTree.

Presumably, this would be possible for BruteForce as well, although maybe computationally more expensive.

### Describe alternatives you've considered, if relevant

Alternatively, I showed how users would loop through samples just to pass in a different radius. Since the API of KDTree and BallTree inherently have this ability, it seems natural to allow `radius` in `NearestNeighbors.radius_neighbors` to be an array.

### Additional context

I am happy to submit a PR if this is deemed relevant.
user: adam2392
issue_25963: state: open
title: RFC 0% tolerance for Codecov patch
body: We currently have a 1% tolerance on Codcov for the project and for each patch.
Although having a tolearance on the project seems legit, we usually target a 100% coverage on PRs.
The results is that sometimes all checks are green and you don't realize that you have uncovered lines unless you explicitly open the codcov tab.

With @glemaitre we propose to set a 0% tolerance for the PRs. When there are lines that we know we can't cover, because it can only be triggered by a specific architecture or by a specific version of a dependency for instance, then we explicitly add a ``# pragma: no cover``.
user: jeremiedbb
issue_25948: state: open
title: FIX minimum norm solution of unpenalized ridge / OLS when fit_intercept=True
body: Fixes: #22947.
Related to: #22910.

The existing test assumes that we should recover the minimum norm solution to the OLS problem where the intercept fitting is implemented by adding an extra constant feature and its matching coef.

However this does not hold. In particular, the intercept component should **not** contribute to the coef norm in the least norm solution as explained in this new note added to the documentation. Furthermore this justifies our pre-centering strategy in `_preprocess_data` and generic `_set_intercept` in all linear regression models with a least squares data-fit term, whatever the regularization term.

TODO:

- [x] document why solving for penalized linear regression with intercept is equivalent to fit on centered data without intercept;
  - [x] find the correct derivation for the underdetermined case
  - [ ] decide where to put this (in the doc or link to an external resource?)
- [ ] update the `ols_ridge_dataset` fixture accordingly and cross-link to the doc;
  - [x] update the underdetermined case (`wide`)
  - [ ] update the ovderderdetermined case (`long`)?
- [ ] make sure the updated tests pass for all ridge solvers;
  - [x] for `wide` and `fit_intercept=True`
  - [ ] for `wide` and `fit_intercept=False`
  - [ ] for `long` and `fit_intercept=True`
  - [ ] for `long` and `fit_intercept=False`
- [x] add inline comments to `_preprocess_data` and `_set_intercept` because this trick was not trivial to me and it took me a while to fully grasp all its implications;
- [ ] add a new test to check that the Ridge solution converges to the minimum `coef_` norm solution for OLS when alpha -> 0 for all solvers.
user: ogrisel
issue_25939: state: open
title: ENH add from_cv_results in RocCurveDisplay
body: Add a method `from_cv_results` in `RocCurveDisplay` to plot several ROC curves at once obtained from the cross-validation.

### Discussions

In terms of design, I created a class handling multiple single-curve displays. An alternative is to alterate `RocCurveDisplay` to accept a list of `ndarray` for the `fpr`, `tpr` and `thresholds`. However, it makes a relatively complex change in the `plot` method as well as adding new attributes as `lines_` in conjunction with `line_`.

The code becomes difficult to read similar to `PartialDependenceDisplay` and the reason is to handle multiple and single curves in the same object.

The reasoning here is to expose `from_cv_results` directly from `RocCurveDisplay` but return a `MultiRocCurveDisplay` that can use the information stored in each underlying display to make the plotting.

It is the based design that we come up with @ogrisel so far.
user: glemaitre
issue_25929: state: open
title: Visual improvements for ROC and precision-recall plots
body: ### Describe the workflow you want to enable

Hello,

I would like to suggest the following improvements:

1. Add x and y axis limit: [0, 1], in sklearn axes currently start at ~-0.1
2. Modify the plotting frame: either remove the top and right lines to see the curve better when values are close to 1, or plot the frame with a dotted line
3. Fix aspect ratio to squared, since the two axes are the same scale.
4. Add title

### Describe your proposed solution

Hello,

While working on ROC and precision-recall plots, I came up with the following suggestions:

1. Add x and y axis limit: [0, 1], in sklearn axes currently start at ~-0.1
2. Modify the plotting frame: either remove the top and right lines to see the curve better when values are close to 1, or plot the frame with a dotted line
3. Fix aspect ratio to squared, since the two axes are the same scale.
4. Add title

```
        ax.set_xlim((-0.01, 1.01))
        ax.set_ylim((-0.01, 1.01))
        for s in ['right', 'left', 'top', 'bottom']:
            ax.spines[s].set_visible(False)
        ax.axvline(1, color='black', linewidth=0.5, ls='dotted')
        ax.axhline(1, color='black', linewidth=0.5, ls='dotted')
        ax.axvline(0, color='black', linewidth=0.5, ls='dotted')
        ax.axhline(0, color='black', linewidth=0.5, ls='dotted')
        ax.set_aspect(1)
        ax.set_title(f'Precision-Recall Curve')
```

### Example 1 Edge case
#### PR
Before:
![image](https://user-images.githubusercontent.com/1184396/226746221-40b40d56-74a4-49c1-82aa-4a3f3366abe2.png)

After:
![image](https://user-images.githubusercontent.com/1184396/226746245-3e0a0ffd-28f0-4851-8a41-04629750bd2c.png)

#### ROC
Before:
![image](https://user-images.githubusercontent.com/1184396/226746270-ecc2a1ce-48dc-4682-8acd-86a4d06dfcf0.png)

After:
![image](https://user-images.githubusercontent.com/1184396/226746289-8ba8c906-f8f8-4d74-9269-b12c16d339fa.png)

### Example 2 - Edge case

#### PR
Before:
![image](https://user-images.githubusercontent.com/1184396/226745897-de69cf76-f520-499f-b8b6-4e9ede9e7ba2.png)

After:
![image](https://user-images.githubusercontent.com/1184396/226745936-5fc34fff-eff3-4a70-8b99-b7c731984723.png)

#### ROC
Before:
![image](https://user-images.githubusercontent.com/1184396/226745998-d1788d4a-fd0a-4293-bef5-170a71c0a098.png)

After:
![image](https://user-images.githubusercontent.com/1184396/226746031-e93dff40-2487-4776-a5b8-baeef9a36e5d.png)



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: stephanecollot
issue_25922: state: open
title: Make complex numbers check optional in `BaseEstimator._validate_and_set_fit_params`
body: ### Describe the workflow you want to enable

I recently extended my library [PySR](https://github.com/MilesCranmer/PySR) to allow for complex `X` and `y` arrays. However, the function `BaseEstimator._validate_and_set_fit_params` has no way of disabling the real number assertion. The function `_ensure_no_complex_data(array)` in `utils/validation.py` cannot be disabled. Thus, the only way for me to get my library working is to monkey patch this function to skip the validation altogether.

### Describe your proposed solution

I would like there to be a parameter for `_validate_and_set_fit_params` that turns off the complex number check, just as there is also a way to turn off the 2D array check, etc.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Here is where the call takes place in my `PySRRegressor` object: https://github.com/MilesCranmer/PySR/blob/ab9ae608a74ea35557c2011e1f1218d300f18a2e/pysr/sr.py#L1346. This object inherits `MultiOutputMixin, RegressorMixin, BaseEstimator`.

Here is an example call and the error I see:

```python
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In [9], line 1
----> 1 model.fit(X, y)

File ~/Documents/PySR/pysr/sr.py:1737, in PySRRegressor.fit(self, X, y, Xresampled, weights, variable_names)
   1733 self._setup_equation_file()
   1735 mutated_params = self._validate_and_set_init_params()
-> 1737 X, y, Xresampled, weights, variable_names = self._validate_and_set_fit_params(
   1738     X, y, Xresampled, weights, variable_names
   1739 )
   1741 if X.shape[0] > 10000 and not self.batching:
   1742     warnings.warn(
   1743         "Note: you are running with more than 10,000 datapoints. "
   1744         "You should consider turning on batching (https://astroautomata.com/PySR/options/#batching). "
   (...)
   1749         "More datapoints will lower the search speed."
   1750     )

File ~/Documents/PySR/pysr/sr.py:1346, in PySRRegressor._validate_and_set_fit_params(self, X, y, Xresampled, weights, variable_names)
   1344     weights = check_array(weights, ensure_2d=False)
   1345     check_consistent_length(weights, y)
-> 1346 X, y = self._validate_data(X=X, y=y, reset=True, multi_output=True)
   1347 self.feature_names_in_ = _check_feature_names_in(self, variable_names)
   1348 variable_names = self.feature_names_in_

File ~/venvs/main/lib/python3.10/site-packages/sklearn/base.py:596, in BaseEstimator._validate_data(self, X, y, reset, validate_separately, **check_params)
    594         y = check_array(y, input_name="y", **check_y_params)
    595     else:
--> 596         X, y = check_X_y(X, y, **check_params)
    597     out = X, y
    599 if not no_val_X and check_params.get("ensure_2d", True):

File ~/venvs/main/lib/python3.10/site-packages/sklearn/utils/validation.py:1074, in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
   1069         estimator_name = _check_estimator_name(estimator)
   1070     raise ValueError(
   1071         f"{estimator_name} requires y to be passed, but the target y is None"
   1072     )
-> 1074 X = check_array(
   1075     X,
   1076     accept_sparse=accept_sparse,
   1077     accept_large_sparse=accept_large_sparse,
   1078     dtype=dtype,
   1079     order=order,
   1080     copy=copy,
   1081     force_all_finite=force_all_finite,
   1082     ensure_2d=ensure_2d,
   1083     allow_nd=allow_nd,
   1084     ensure_min_samples=ensure_min_samples,
   1085     ensure_min_features=ensure_min_features,
   1086     estimator=estimator,
   1087     input_name="X",
   1088 )
   1090 y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
   1092 check_consistent_length(X, y)

File ~/venvs/main/lib/python3.10/site-packages/sklearn/utils/validation.py:866, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
    858         raise ValueError(
    859             "Complex data not supported\n{}\n".format(array)
    860         ) from complex_warning
    862 # It is possible that the np.array(..) gave no warning. This happens
    863 # when no dtype conversion happened, for example dtype = None. The
    864 # result is that np.array(..) produces an array of complex dtype
    865 # and we need to catch and raise exception for such cases.
--> 866 _ensure_no_complex_data(array)
    868 if ensure_2d:
    869     # If input is scalar raise error
    870     if array.ndim == 0:

File ~/venvs/main/lib/python3.10/site-packages/sklearn/utils/validation.py:568, in _ensure_no_complex_data(array)
    561 def _ensure_no_complex_data(array):
    562     if (
    563         hasattr(array, "dtype")
    564         and array.dtype is not None
    565         and hasattr(array.dtype, "kind")
    566         and array.dtype.kind == "c"
    567     ):
--> 568         raise ValueError("Complex data not supported\n{}\n".format(array))

ValueError: Complex data not supported
[[-2.02335203-0.13285217j -1.14560511+0.36383775j -0.0388407 -0.56413576j
   0.22136457-0.65906156j  1.31305444+0.33418455j]
 [-0.8686053 -0.20667045j  1.44976451-0.91981686j -0.54253912-0.28040131j
   2.77618372+1.37670423j  1.24366589+1.23424509j]
 [-1.18890458-0.68708739j  1.08839574+0.3281328j  -0.39634625-0.5159068j
   0.21911703+0.87334088j  0.19518873-0.24635224j]
 ...
 [-0.5148545 +0.7814871j   1.44093924-0.65092289j  2.05879756-0.53332935j
   0.55355862+0.2447758j  -0.59905953-0.41042992j]
 [-0.2645599 +2.00658263j  0.27197639+1.48677505j  0.34973374-0.80205851j
   0.39487604-0.26306846j  0.10859811-0.41198692j]
 [ 0.08583031-0.84073204j -0.84911584-0.46094548j -0.20988759-0.66359283j
   0.0835439 -0.73089652j  1.66839496+1.26268374j]]
```
user: MilesCranmer
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_25896: state: open
title: Support other dataframes like polars and pyarrow not just pandas
body: ### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler, sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.
user: lorentzenchr
issue_25894: state: open
title: TST test_impute use global random seed
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Towards https://github.com/scikit-learn/scikit-learn/issues/22827

#### What does this implement/fix? Explain your changes.
set global random seed for impute tests

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Veghit
issue_25891: state: open
title: TST random seed global /svm/tests/test_svm.py
body: Reference Issues/PRs
Towards https://github.com/scikit-learn/scikit-learn/issues/22827

What does this implement/fix? Explain your changes.
using random global seed for sklearn/svm/tests/test_svm.py to enable better testing.

Any other comments?
tests pass without logic changes. Decided to use global seed also for the test_svc_nonfinite_params test as it does not take much time and has some potential issues even though it is a simple test.
user: Veghit
issue_25888: state: open
title: PERF `PairwiseDistancesReductions` subsequent work
body: `PairwiseDistancesReductions` have been introduced as a hierarchy of Cython classes to implement back-ends of some scikit-learn algorithms.

Initial work was listed in #22587.

:bulb: See [this presentation](https://docs.google.com/presentation/d/1RwX_P9lnsb9_YRZ0cA88l3VoEYhiMndQYoKLLF_0Dv0/edit?usp=sharing) for a better understanding of the design of `PairwiseDistancesReductions`.

:bulb: **Before working on improving performance, one must profile the current execution of algorithms to see if there are any substantial benefits.**

Subsequent possible work includes:

 - [ ] #26983 
 - [ ] #25097
 - [ ] Support `"precomputed"` distances
 - Implement back-end for other `Estimators` or `Transformers`, in particular, introduce a specialised back-end for which would remove the costly sequential portion around the current call to `kneighbors` or `radius_neighbors` for (non-complete list):
 	- [ ] computing the MST from the data matrix (`mst_from_data_matrix`) see https://github.com/scikit-learn/scikit-learn/pull/25656#pullrequestreview-1326267314
 	- [ ] computing the MST linkage ([`mst_linkage_core`](https://github.com/scikit-learn/scikit-learn/blob/2ab1d81e167512719b6272d4ddf286bba5d3d232/sklearn/cluster/_hierarchical_fast.pyx#L428-L506))
 	- [x] `LocalOutlierFactor` ([no further optimization possible](https://github.com/scikit-learn/scikit-learn/pull/26316))
 	- [ ] `KNeighbors*.predict*`
 	   - [x] #24076 by @Micky774
 	   - [ ] #25888 by @OmarManzoor
 	   - [ ] Implement multi-output support (WIP [here](https://github.com/Micky774/scikit-learn/pull/3))
 	   - [ ] Support `"distance"` weighting
 	- [ ] `RadiusNeighbors*.predict*`
 	   - [x] #26828
 	   - [ ] Implement Euclidean specialization
 	   - [ ] Implement multi-output support
 	   - [ ] Support `"distance"` weighting
 	- some part of `KMeans`
 - [ ] Force the use of `PairwiseDistancesReductions` for F-contiguous arrays
   - use a scikit-learn configuration entry to accept and convert F-contiguous arrays to C-contiguous arrays
 - [ ] Release the GIL before calling `parallel_on_{X|Y}` instead of releasing it in those methods
    - :bulb: I once have tried and I don't think this is possible.
 - [ ] Integrate `mimalloc` to have proper memory allocation for multi-threaded implementations:
    - :bulb: Changing the default implementation of `malloc(1)` might come with unexpected changes and might break the whole ecosystem. Also maintaining it might be costly.
 - [ ] Tests improvements ideas:
   - https://github.com/scikit-learn/scikit-learn/pull/27281#issuecomment-1704334881
   - parametrize old tests for public API backed by `PairwiseDistancesReductions`?
   - systematically test public API on combinations of sparse and dense datasets?
 - [ ] Extend this design for a new back-end for [`pairwise_kernels`](https://github.com/scikit-learn/scikit-learn/blob/7d1c31892d34e6456c467c3d538ecb0ec4e01c35/sklearn/metrics/pairwise.py#L2116)-based interfaces
user: jjerphan
labels: Meta-issue: General issue associated to an identified list of tasks
issue_25859: state: open
title: How to early stop in GradientBoostingClassifer?
body: How to track the model performance on an eval set that is provided from outside and early stop the tree building based upon the result?

Currently there is the option of ```validation_fraction``` along with ```n_iter_no_change``` available in the implementation
The issue with that approach:
1. Cannot use k-fold cross-validation.
2. Canont use custom metrics
Currently I solve it using the following, which is kind of hacky

```python
#create a gradient booster
gbc = GradientBoostingClassifier()

#define the metric function that you want to use to early stopping
def accuracy(y_true, y_preds):
    return #return the metric output here
#This class along with the monitor argument will enable early stopping
class early_stopping_gbc():
    def __init__(self, accuracy, eval_set, early_stopping_rounds = 20):
        self.accuracy = accuracy
        self.x_val = eval_set[0]
        self.y_val = eval_set[1]
        self.best_perf = 0.
        self.counter = 0
        self.early_stopping_rounds = early_stopping_rounds
    def __call__(self,i, model, local_vars):
        for counter, preds in enumerate(model.staged_predict_proba(self.x_val)):
            if counter == i:
                break
        acc = self.accuracy(self.y_val,preds[:,1])
        if acc > self.best_perf:
            self.best_perf = acc
            self.counter = 0
        else:
            self.counter += 1
        return self.counter > self.early_stopping_rounds

#Run gradient booster with early stopping on 20 rounds
gbc.fit(X_train,y_train, monitor = early_stopping_gbc(accuracy, [X_val,y_val], early_stopping_rounds = 20))
```

user: Nitinsiwach
labels: Needs Decision - API: Requires decision regarding API
issue_25856: state: open
title: Sampling uncertainty on precision-recall and ROC curves
body: ### Describe the workflow you want to enable

We would like to add the possibility to plot sampling uncertainty on precision-recall and ROC curves.


### Describe your proposed solution

We (@mbaak, @RUrlus, @ilanfri and I) published a paper in [AISTAT 2023](http://aistats.org/aistats2023/accepted.html) called [Pointwise sampling uncertainties on the Precision-Recall curve](https://github.com/RUrlus/ModelMetricUncertaintyResearch/tree/sklearn_pull_request/notebooks/pr_ellipse_validation/1411.pdf), where we compared multiple methods to compute and plot them.

We found out that a great way to compute them is to use profile likelihoods based on Wilks’ theorem.
It consists of the following steps:

1. Get the curve
2. Get the confusion matrix of each point of the curve
3. For each observed point of the curve, estimate a surrounding 6 (i.e. more than the desired number) sigmas uncertainty grid rectangle (based on first-order approximation of the covariance matrix, with the bivariate normal distribution assumption) 
4. For each of these hypothesis point in the grid, compute the test static with the observed point, called the profile log likelihood ratio (using the fact that the confusion matrix follows a multinomial distribution).
5. Plot the 3 sigmas contour (i.e. isoline) for the observed points (using Wilks’ theorem stating that the profile log likelihood ratio is described asymptotically by a chi2 distribution)


We have a minimal pure Python implementation:
https://github.com/RUrlus/ModelMetricUncertaintyResearch/blob/sklearn_pull_request/notebooks/pr_ellipse_validation/demo_ROC_PR_curves_sklearn_pull_request.ipynb 

And a C++ implementation: the paper is supported by our package [ModelMetricUncertainty](https://github.com/RUrlus/ModelMetricUncertainty) which has a C++ core with, optional, OpenMP support and Pybind11 bindings. Note that this package contains much more functionality than the above notebook. The core is binding agnostic allowing a switch to Cython if needed. Upside is that it is much faster (multiple orders) than the above Python implementation at the cost of complexity.

The pure Python implementation would look like this:
![image4](https://user-images.githubusercontent.com/1184396/225160909-f03bc592-ffb0-4b04-8f6a-eadd889c65a1.png)

I’m also suggesting other visual improvements:
1. Add x and y axis limit: [0, 1], in sklearn axes currently start at ~-0.1
2. Modify the plotting frame: either remove the top and right lines to see the curve better when values are close to 1, or plot the frame with a dotted line
3. Fix aspect ratio to squared, since the two axes are the same scale.

With those it can look like this:
![download-3](https://user-images.githubusercontent.com/1184396/225161339-eb794d06-073e-4c4b-932e-618227e4925a.png)
![download-2](https://user-images.githubusercontent.com/1184396/225161357-e56c8077-271b-4ae2-b92d-1770c17f538c.png)

Remark: I set the contour color to lightblue, let me know if it is fine.

We need to align on the API integration. I suggest adding some parameters in `PrecisionRecallDisplay` and in `RocCurveDisplay` called:

- `uncertainty=True` to enabel plot uncertainty band (or `plot_uncertainty_style=` ?)
- `uncertainty_n_std=3` to decide how +/- standard deviation the band should be
- `uncertainty_n_bins=100` to decide how fine-grained the band should be (see remark about running time)

### Describe alternatives you've considered, if relevant

Other ways to compute uncertainties are evaluated in our paper.

We have noticed that there is open pull request on related topic: https://github.com/scikit-learn/scikit-learn/pull/21211
That is great, however cross-validation covers different sources of uncertainties, and has some limitations (a bias is introduced by overlapping training folds, introducing a correlation in the trained models. In addition, this uncertainty depends on the size of a fold, and is likely larger than on the test set, see [ref](https://www.jmlr.org/papers/volume5/grandvalet04a/grandvalet04a.pdf).)


### Additional context

##### Running time discussion
Here is an analysis of the running time of this pure Python method:

<img width="676" alt="image5" src="https://user-images.githubusercontent.com/1184396/225161990-44fae526-e25d-4129-b03f-0ce24e2df530.png">


The execution time depends on the number of points (i.e. thresholds) plotte and on `uncertainty_n_bins`.
With a surrounding grid of `uncertainty_n_bins=100` per point it is fast enough and fine enough.
There is barely any noticeable visual difference between 50 and 100 (or more) points (at least in this example), [see curves](https://github.com/RUrlus/ModelMetricUncertaintyResearch/tree/sklearn_pull_request/notebooks/pr_ellipse_validation/curves).
For let’s say for a 100k set, it is too slow for ROC, because there is much more thresholds, but this is going to be fixed soon here https://github.com/scikit-learn/scikit-learn/pull/24668 . But anyway, in this case, the uncertainties are really small, so plotting them doesn’t really make.
user: stephanecollot
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_25824: state: open
title: FastOPTICS in `sklearn.cluster`
body: ### Describe the workflow you want to enable

Is there a roadmap to add the FastOPTICS algorithm, [1], to the `sklearn.cluster` code base that already supports OPTICS?

[1] 2013, J. Schneider, M. Vlachos, _Fast Parameterless Density-based Clustering via Random Projections 

### Describe your proposed solution

The solution would be to combine what has already been done for the base OPTICS algorithm, combined with the existing code base for random projections and the Johnson-Lindenstrauss bound in `sklearn.random_projection`, to implement FastOPTICS.

The [implementation](https://github.com/elki-project/elki/blob/master/elki-clustering/src/main/java/elki/clustering/optics/FastOPTICS.java) in the data mining library ELKI (albeit in Java) could be used as an inspiration.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: DiTo97
labels: Needs Decision: Requires decision
issue_25812: state: open
title: Possible performance improvements for kd-tree or ball-tree with multi-threading at fit time
body: As reported here: https://skops.readthedocs.io/en/latest/auto_examples/plot_intelex.html

For small `n_features`, `KNeighborsClassifier` uses either BallTree of KD-tree algorithms. At this time, scikit-learn builds this tree sequentially.

We could recursively build the branches in parallel using a `ThreadPoolExecutor` instance and call submit for each children of a given node. Note that each node splitting operation (starting from the root) is one Cython function call that releases the GIL, so using thread-based parallelism should be efficient.

The second speed-up reported in plot_intelex.html above (at inference time) is mostly linked to the fact that scikit-learn `KNeighborsClassifier` does not enable parallel neighbors queries with the default value of `n_jobs` when using the kd-tree or ball-tree algorithms.

From a user's point of view, this is not ideal either: when using the brute-force algorithm, we rely on OpenMP and automatically use as many threads as CPUs available.
user: ogrisel
issue_25807: state: open
title:  [MRG] FEA ICE lines individually colored by feature values
body: #### Reference Issues/PRs
<!--
No open Issue (that we're aware of)
-->


#### What does this implement/fix? Explain your changes.
This PR introduces a new feature, which is an extension of the Individual Conditional Expectation (ICE) Plots.
So far a single color for the ICE lines could be chosen by the user (in this case blue):
```
display = PartialDependenceDisplay.from_estimator(
    hgbdt_model,
    X_train,
    **features_info,
    ax=ax,
    **common_params,
)
```

![standard](https://user-images.githubusercontent.com/34746570/224180938-62b2e28b-2077-41e2-9a17-33be45246656.png)


This extension allows coloring each line according to the values of another feature. For this, the user just has to pass a list of values. In this case, we use the discrete "workingday" feature from the bike rental dataset. True indicates workingday, false the weekend. This is as easy as just passing the series as the "color" parameter in the ice_lines_kw:
```
display = PartialDependenceDisplay.from_estimator(
    hgbdt_model,
    X_train,
    **features_info,
    ax=ax,
    **common_params,
    ice_lines_kw={"color": X_train["workingday"], "palette": "Paired_r"},
)
```

![workingday](https://user-images.githubusercontent.com/34746570/224181667-568db76c-4453-4a71-ac07-3e5d68dbf052.png)

It also works with continuous features, for example, the hour of the bike rental:
```

display = PartialDependenceDisplay.from_estimator(
    hgbdt_model,
    X_train,
    **features_info,
    ax=ax,
    **common_params,
    ice_lines_kw={"color": X_train["hour"], "palette": "cool"},
)
```

![hour](https://user-images.githubusercontent.com/34746570/224181707-eb05f4b2-0e17-4103-b769-95b385332d6b.png)

The advantage of this is that we get a more detailed picture and can visualize dependencies between the features. 
In the first example (all lines in blue) all the plot tells us is that roughly half the lines show a sharp increase in bike rentals (Partial dependence) once the temperature rises above 18°, but it is unclear why some lines show this increase and others not.

With the new functionality, it is possible to dig deeper: The increase in bike rentals almost exclusively happens on the weekend. Possibly because during the week people need to get to and back from work independent of the temperature, whereas on the weekend warmer weather increases the number of bike rentals.


#### Any other comments?
The example and data is taken from here:
https://scikit-learn.org/stable/auto_examples/inspection/plot_partial_dependence.html

The plots above are also added to this page. The documentation renders well when I tested it, but the output doesn't show up.

I can't understand why the checks regarding matplotlib.colormaps fail, I can run it locally. This was also an attribute that was suggested a DeprecationWarning in matplotlib
user: lurue101
issue_25799: state: open
title: Incorrect results in kernel density estimation (```KernelDensity```) when the kernel width is small
body: ### Describe the bug

I have met an issue when using the class ```sklearn.neighbors.KernelDensity```. The **kernel density estimation (KDE)** works unusually when the kernel width is very small (less than 0.01),e.g., 0.00001. The output kde scores from the function ```score_samples``` are all big constants and almost the same, which is obviously very strange, not likely to be the correct densities. 
For example in the sample code, I generated 1000 Gaussian distributed data, used it to fit the KDE model with a small kernel width (0.00001), and again evaluated the model on the same data. Then, the outputs from ```score_samples``` are almost 3.6862 and the prob are 39.8942 after applying a ```np.exp()``` on top of it. The smaller the kernel width is, the worse KDE scores would be.  
So is the KDE itself still correct when the kernel width is very small, or perhaps is there any issue with the normalization in the implementation? Thank you very much for checking this issue and the help!

### Steps/Code to Reproduce

```python
from sklearn.neighbors import KernelDensity
import numpy as np

# generate training data
rand = np.random.RandomState(seed=100)
data = rand.normal(0, 1, 1000).reshape(-1, 1)

# fit the training data and evaluate on it
# kde scores tend to be probmatic when bandwidth is very small, e.g., 0.00001
model = KernelDensity(bandwidth=0.00001, kernel='gaussian')
model.fit(data)
log_dens = model.score_samples(data)
print(log_dens)
print(np.exp(log_dens))

```

### Expected Results

The results shouldn't be like such big constants and almost the same.

### Actual Results

When the kernel width is 0.00001, the output from ```score_samples``` for the first 20 data:
 ``` 
[3.68623165 3.68623165 3.68623165 3.68623165 3.68623165 3.68623165
 3.68623165 3.68623165 3.68623165 3.68623165 3.68623165 3.68623165
 3.68623165 3.68623165 3.68623165 3.68623165 3.68623165 3.68623165
 3.97824858 3.68623165]
```
After applying ```np.exp()``` on top of the above results:
```
[39.89422804 39.89422804 39.89422804 39.89422804 39.89422804 39.89422804
 39.89422804 39.89422804 39.89422804 39.89422804 39.89422804 39.89422804
 39.89422804 39.89422804 39.89422804 39.89422804 39.89422804 39.89422804
 53.42338568 39.89422804]
```
The results is even worse when the kernel width is smaller, e.g., when the kernel width is 0.000001, the results would be:
Output from ```score_samples```:
```
[5.98881675 5.98881675 5.98881675 5.98881675 5.98881675 5.98881675
 5.98881675 5.98881675 5.98881675 5.98881675 5.98881675 5.98881675
 5.98881675 5.98881675 5.98881675 5.98881675 5.98881675 5.98881675
 5.98881675 5.98881675]
```
After ```np.exp()```:
```
[398.9422804 398.9422804 398.9422804 398.9422804 398.9422804 398.9422804
 398.9422804 398.9422804 398.9422804 398.9422804 398.9422804 398.9422804
 398.9422804 398.9422804 398.9422804 398.9422804 398.9422804 398.9422804
 398.9422804 398.9422804]
```


### Versions

```shell
System:
    python: 3.9.16 (main, Jan 11 2023, 10:02:19)  [Clang 14.0.6 ]
executable: /Users/tongtong/opt/anaconda3/envs/torch/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.2.1
          pip: 22.3.1
   setuptools: 65.6.3
        numpy: 1.23.5
        scipy: 1.10.1
       Cython: None
       pandas: 1.5.2
   matplotlib: 3.6.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /Users/tongtong/opt/anaconda3/envs/torch/lib/python3.9/site-packages/sklearn/.dylibs/libomp.dylib
        version: None
    num_threads: 8

       user_api: blas
   internal_api: mkl
         prefix: libmkl_rt
       filepath: /Users/tongtong/opt/anaconda3/envs/torch/lib/libmkl_rt.1.dylib
        version: 2021.4-Product
threading_layer: intel
    num_threads: 8

       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /Users/tongtong/opt/anaconda3/envs/torch/lib/libomp.dylib
        version: None
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /Users/tongtong/opt/anaconda3/envs/torch/lib/python3.9/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.18
threading_layer: pthreads
   architecture: Nehalem
    num_threads: 8
```

user: TongtongFANG
issue_25789: state: open
title: API Gaussian Process: change default behavior of `sample_y`
body: The `random_state` argument was set to `0` forceing `sample_y` to always return the same gaussian process samples.
This is contrary to the expected behavior most people have from working with random number generating libraries such as numpy and scipy. Changing the default parameter to `None` will make the behavior of `sample_y` conform to the usual expectation a user.

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes #25767
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
Changes default parameter `random_state` from `GaussianProcessRegressor` from `0` to `None` to make it behave like a random
number generator (RNG) by default, conforming with all other RNG APIs.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: ominusliticus
labels: Waiting for Second Reviewer: First reviewer is done, need a second one!
issue_25771: state: open
title: maximal information coefficient for feature selection
body: ### Describe the workflow you want to enable

maximal information coeffcient

### Describe your proposed solution

maximal information coeffcient

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: leeauk21
issue_25767: state: open
title: Change default behavior of `GaussianProcessRegressor.sample_y`'s `random_state` variable.
body: ### Describe the workflow you want to enable

I think it is misleading to have the `sample_y` method to default to reproduce the same sample every time it is called.
The default value for `random_state` should be `None`, and the documentation `sample_y` should state that `None` ensures random behavior.



### Describe your proposed solution

Please see above box

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: ominusliticus
issue_25766: state: open
title: 1.2.1: cannot build documentation without installing module
body: ### Describe the bug

Looks like something is wrong and I cannot build docuemtation without installing module.


### Steps/Code to Reproduce

N/A

### Expected Results

It should be possible to build documentation out of only what is in source tree a d withoit have installed module.

### Actual Results

<details>

```console
++ ls -1d lib.linux-x86_64-cpython-38
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/build/lib.linux-x86_64-cpython-38
+ PBR_VERSION=1.2.1
+ PDM_PEP517_SCM_VERSION=1.2.1
+ SETUPTOOLS_SCM_PRETEND_VERSION=1.2.1
+ /usr/bin/sphinx-build -n -T -b man doc build/sphinx/man
Running Sphinx v6.1.3

Traceback (most recent call last):
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__check_build/__init__.py", line 48, in <module>
    from ._check_build import check_build  # noqa
ModuleNotFoundError: No module named 'sklearn.__check_build._check_build'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 351, in eval_config_file
    exec(code, namespace)  # NoQA: S102
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/doc/conf.py", line 20, in <module>
    from sklearn.externals._packaging.version import parse
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__init__.py", line 81, in <module>
    from . import __check_build  # noqa: F401
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__check_build/__init__.py", line 50, in <module>
    raise_build_error(e)
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__check_build/__init__.py", line 31, in raise_build_error
    raise ImportError(
ImportError: No module named 'sklearn.__check_build._check_build'
___________________________________________________________________________
Contents of /home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__check_build:
__init__.py               _check_build.pyx          _check_build.c
__pycache__
___________________________________________________________________________
It seems that scikit-learn has not been built correctly.

If you have installed scikit-learn from source, please do not forget
to build the package before using it: run `python setup.py install` or
`make` in the source directory.

If you have used an installer, please check that it is suited for your
Python version, your operating system and your platform.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/sphinx/cmd/build.py", line 279, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
  File "/usr/lib/python3.8/site-packages/sphinx/application.py", line 202, in __init__
    self.config = Config.read(self.confdir, confoverrides or {}, self.tags)
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 173, in read
    namespace = eval_config_file(filename, tags)
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 364, in eval_config_file
    raise ConfigError(msg % traceback.format_exc()) from exc
sphinx.errors.ConfigError: There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__check_build/__init__.py", line 48, in <module>
    from ._check_build import check_build  # noqa
ModuleNotFoundError: No module named 'sklearn.__check_build._check_build'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 351, in eval_config_file
    exec(code, namespace)  # NoQA: S102
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/doc/conf.py", line 20, in <module>
    from sklearn.externals._packaging.version import parse
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__init__.py", line 81, in <module>
    from . import __check_build  # noqa: F401
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__check_build/__init__.py", line 50, in <module>
    raise_build_error(e)
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__check_build/__init__.py", line 31, in raise_build_error
    raise ImportError(
ImportError: No module named 'sklearn.__check_build._check_build'
___________________________________________________________________________
Contents of /home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__check_build:
__init__.py               _check_build.pyx          _check_build.c
__pycache__
___________________________________________________________________________
It seems that scikit-learn has not been built correctly.

If you have installed scikit-learn from source, please do not forget
to build the package before using it: run `python setup.py install` or
`make` in the source directory.

If you have used an installer, please check that it is suited for your
Python version, your operating system and your platform.


Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__check_build/__init__.py", line 48, in <module>
    from ._check_build import check_build  # noqa
ModuleNotFoundError: No module named 'sklearn.__check_build._check_build'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 351, in eval_config_file
    exec(code, namespace)  # NoQA: S102
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/doc/conf.py", line 20, in <module>
    from sklearn.externals._packaging.version import parse
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__init__.py", line 81, in <module>
    from . import __check_build  # noqa: F401
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__check_build/__init__.py", line 50, in <module>
    raise_build_error(e)
  File "/home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__check_build/__init__.py", line 31, in raise_build_error
    raise ImportError(
ImportError: No module named 'sklearn.__check_build._check_build'
___________________________________________________________________________
Contents of /home/tkloczko/rpmbuild/BUILD/scikit-learn-1.2.1/sklearn/__check_build:
__init__.py               _check_build.pyx          _check_build.c
__pycache__
___________________________________________________________________________
It seems that scikit-learn has not been built correctly.

If you have installed scikit-learn from source, please do not forget
to build the package before using it: run `python setup.py install` or
`make` in the source directory.

If you have used an installer, please check that it is suited for your
Python version, your operating system and your platform.
```
</details>

### Versions

```shell
1.2.1
```

user: kloczek
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_25755: state: open
title: Figure is shown incorrectly
body: ### Describe the issue linked to the documentation

Second part of the [figure](https://scikit-learn.org/stable/modules/gaussian_process.html#dot-product-kernel) doesn't show sampled functions

### Suggest a potential alternative/fix

_No response_
user: Mafmax
labels: Needs Investigation: Issue requires investigation
issue_25750: state: open
title: Differences in scalar vs vectorized predictions with `GaussianProcessRegressor`
body: ### Describe the bug

I would expect that calling `GaussianProcessRegressor.predict(X)` with a single X matrix, or with repeated scalar evaluations rows of X should would give nearly the same result, but they don't.

An example is given below.

### Steps/Code to Reproduce

```python
import numpy as np
import pandas as pd
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import DotProduct, RBF
from sklearn.model_selection import train_test_split


data_url = "http://lib.stat.cmu.edu/datasets/boston"
raw_df = pd.read_csv(data_url, sep="\s+", skiprows=22, header=None)
data = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])
target = raw_df.values[1::2, 2]


def evaluate_gpr_vec_and_scalar(gpr, X):
    res_vector = gpr.predict(X).squeeze()
    res_scalar = np.array([gpr.predict(xi.reshape(1, -1)) for xi in X]).squeeze()
    return res_scalar, res_vector

# load/split data
X_train, X_test, y_train, y_test = train_test_split(data, target, random_state=0)

# train
gpr = GaussianProcessRegressor(DotProduct() + RBF(), alpha=1.)
gpr.fit(X_train, y_train)

# predict scalar and vector
pred_scalar, pred_vector = evaluate_gpr_vec_and_scalar(gpr, X_test)
err = pred_scalar - pred_vector
print(err.max())
```

### Expected Results

I would expect the error to be 0 or small - perhaps around machine `eps`, if anything.

### Actual Results

`2.1609594114124775e-08`


### Versions

```shell
System:
    python: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:41:54) [Clang 13.0.1 ]
executable: /Users/phil/miniconda3/envs/skl/bin/python
   machine: macOS-12.5.1-x86_64-i386-64bit
Python dependencies:
      sklearn: 1.1.1
          pip: 22.3
   setuptools: 65.5.0
        numpy: 1.23.4
        scipy: 1.9.3
       Cython: None
       pandas: 1.5.1
   matplotlib: 3.6.1
       joblib: 1.2.0
threadpoolctl: 3.1.0
Built with OpenMP: True
threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /Users/phil/miniconda3/envs/skl/lib/libopenblasp-r0.3.21.dylib
        version: 0.3.21
threading_layer: openmp
   architecture: Nehalem
    num_threads: 8
       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /Users/phil/miniconda3/envs/skl/lib/libomp.dylib
        version: None
    num_threads: 8
```

user: whophil
labels: Needs Investigation: Issue requires investigation
issue_25740: state: open
title: Verbose for Gaussian Process regression
body: ### Describe the workflow you want to enable

I would like to monitor the progress of my fit function for a Gaussian Process. 

### Describe your proposed solution

if verbose >= 1 print the parameters "theta" and the log marginal likelihood. Maybe offer the option to return the log likelihood with a dict for the tried thetas.
- In general any verbose would be very welcome. To get a rough estimate of the progress. 

Please and thank you.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: ithakis
issue_25716: state: open
title: Is the check of strict convergence in KMeans too expensive for the benefits ?
body: ### Describe the bug

In `KMeans` scikit-learn defines [`strict_convergence`](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/cluster/_kmeans.py#L701) as the event of producing the same label assignments at two successive iterations.

When this happens, it means convergence for both labels and centroids (set aside possible oscillations due to numerical instability, were the iterations to continue).

But checking for strict convergence seems to be somewhat expensive (one loop over the last two label assignments of each sample per iteration), and if the user properly set `tol` it doesn't seem necessary at all ?

Checking for strict convergence seems to really help when `tol==0`. With `tol==0` I've seen cases of endless oscillations around 0 because of numerical instability, but never reaching 0, and finally terminating at `max_iter` iterations.

For the general case, isn't it detrimental to performance though ? one can expect the performance cost to be significant for small dimensions of data, for which an additional pass on a column is marginally more expensive.

So I would maybe suggest the following improvements:

- [ ] enable automatically the strict convergence checks only if `tol==0` (or if `tol` is "very small")
- [ ] maybe expose to the user the choice of enabling strict convergence at each iteration ?


### Versions

```shell
1.3
```

user: fcharras
labels: Needs Investigation: Issue requires investigation
issue_25714: state: open
title: DOC Add demo on parallelization with context manager using different backends
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.

We currently have some very insightful information about parallelism on the [user guide](https://scikit-learn.org/stable/computing/parallelism.html), where we even link to [this document](https://thomasjpfan.github.io/parallelism-python-libraries-design/) from @thomasjpfan. But IMHO a demo using different backends might be the cherry on the cake.

#### Any other comments?

I tried benchmarking on spark but I had several problems of compatibility and did not seem to improve the computing time on a local computer.

This work is a joint effort with @Vincent-Maladiere.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: ArturoAmorQ
issue_25711: state: open
title: SequentialFeatureSelector is not working with ColumnTransformer
body: ### Describe the bug

Please see the code.

### Steps/Code to Reproduce

```python
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer

import pandas as pd
import numpy as np


# dummy data
N = 100
dummy_x = pd.DataFrame(
    np.random.randn(N,3),
    columns = list('abc'),
)

dummy_y = pd.DataFrame(
    np.random.choice([0,1], size= (N,1)),
    columns = ['label'],
)


num_pipeline = Pipeline([
        ('imputer', SimpleImputer(strategy="median")),
        ('std_scaler', StandardScaler()),
    ])

ct_parts = [
                ('num', num_pipeline, [0,1,2]),
]

data_preparation_pipe = ColumnTransformer(ct_parts, remainder='passthrough')


from sklearn.feature_selection import SequentialFeatureSelector
from sklearn.ensemble import GradientBoostingClassifier

model = Pipeline(
    [
        # ('data_prep', num_pipeline),
        ('data_prep', data_preparation_pipe),
        ('ML', GradientBoostingClassifier()),
    ]
)

sfs = SequentialFeatureSelector(
    model,
)

sfs.fit(dummy_x, dummy_y)
```




### Expected Results

No error

### Actual Results

```python-traceback
Traceback (most recent call last):
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3460, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "/tmp/ipykernel_3433582/3663298101.py", line 1, in <module>
    sfs.fit(dummy_x, dummy_y)
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/feature_selection/_sequential.py", line 268, in fit
    new_feature_idx, new_score = self._get_best_new_feature_score(
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/feature_selection/_sequential.py", line 299, in _get_best_new_feature_score
    scores[feature_idx] = cross_val_score(
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 515, in cross_val_score
    cv_results = cross_validate(
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 285, in cross_validate
    _warn_or_raise_about_fit_failures(results, error_score)
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 367, in _warn_or_raise_about_fit_failures
    raise ValueError(all_fits_failed_message)
ValueError: 
All the 5 fits failed.
It is very likely that your model is misconfigured.
You can try to debug the error by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
5 fits failed with the following error:
Traceback (most recent call last):
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/utils/__init__.py", line 416, in _get_column_indices
    idx = _safe_indexing(np.arange(n_columns), key)
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/utils/__init__.py", line 356, in _safe_indexing
    return _array_indexing(X, indices, indices_dtype, axis=axis)
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/utils/__init__.py", line 185, in _array_indexing
    return array[key] if axis == 0 else array[:, key]
IndexError: index 1 is out of bounds for axis 0 with size 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 686, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/pipeline.py", line 401, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/pipeline.py", line 359, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/joblib/memory.py", line 349, in __call__
    return self.func(*args, **kwargs)
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/pipeline.py", line 893, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 142, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/compose/_column_transformer.py", line 724, in fit_transform
    self._validate_column_callables(X)
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/compose/_column_transformer.py", line 426, in _validate_column_callables
    transformer_to_input_indices[name] = _get_column_indices(X, columns)
  File "/software/anaconda3/envs/TOSC_ML/lib/python3.10/site-packages/sklearn/utils/__init__.py", line 418, in _get_column_indices
    raise ValueError(
ValueError: all features must be in [0, 0] or [-1, 0]
```

### Versions

```shell
System:
    python: 3.10.9 | packaged by conda-forge | (main, Feb  2 2023, 20:20:04) [GCC 11.3.0]
executable: /software/anaconda3/envs/TOSC_ML/bin/python
   machine: Linux-4.18.0-305.3.1.el8.x86_64-x86_64-with-glibc2.28

Python dependencies:
      sklearn: 1.2.1
          pip: 23.0
   setuptools: 67.3.2
        numpy: 1.23.5
        scipy: 1.10.0
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.7.0
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /software/anaconda3/envs/TOSC_ML/lib/libopenblasp-r0.3.21.so
        version: 0.3.21
threading_layer: pthreads
   architecture: Zen
    num_threads: 128

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /software/anaconda3/envs/TOSC_ML/lib/libgomp.so.1.0.0
        version: None
    num_threads: 128
```

user: Crispy13
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_25707: state: open
title: ENH HistGradientBoosting estimators should have `.feature_importances_` attribute
body: ### Describe the workflow you want to enable

The practicality of using the `.feature_importances_` attribute to superficially analyze global explanations of your classifier is really useful, IMHO.

Right after evaluating the model, running something like
```python
pd.DataFrame(model.feature_importances_, index=X_train.columns, columns=["feat_imp"]).sort_values("feat_imp", ascending=False)
```
is the first thing I do.

Furthermore, some variable selection strategies assume that your estimator has this attribute, and it seems suboptimal to me to deprive ourselves of using `HistGradientBoosting` with these techniques.
```python
from sklearn.feature_selection import SelectFromModel
from sklearn.ensemble import HistGradientBoostingClassifier
from lightgbm import LGBMClassifier

X = [[ 0.87, -1.34,  0.31 ],
     [-2.79, -0.02, -0.85 ],
     [-1.34, -0.48, -2.55 ],
     [ 1.92,  1.48,  0.65 ]]
y = [0, 1, 0, 1]
selector = SelectFromModel(estimator=LGBMClassifier()).fit(X, y)
selector.get_support()
>>> array([ True,  True,  True])

selector = SelectFromModel(estimator=HistGradientBoostingClassifier()).fit(X, y)
selector.get_support()
>>> ValueError: when `importance_getter=='auto'`, the underlying estimator HistGradientBoostingClassifier should have `coef_` or `feature_importances_` attribute. Either pass a fitted estimator to feature selector or call fit before calling transform.
```

I understand the concerns raised by @ogrisel's [comment in the original PR that implemented HistGradientBoosting](https://github.com/scikit-learn/scikit-learn/pull/12807#issuecomment-462802273), but I believe we have more points in favor of its implementation than its omission. I'd love to hear your thoughts. :)

### Describe your proposed solution

_No response_

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: vitaliset
labels: Needs Decision - API: Requires decision regarding API
issue_25689: state: open
title: FIX Multioutput estimators fail to return raw values
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Fixes #25666. See also #18343.

#### What does this implement/fix? Explain your changes.

The goal of the multioutput="raw_values" parameter in some metrics is to be able to inspect the individual scores of a multioutput metaestimator, but the `_score` function in `_validation.py` expects a number, not the array it actually outputs.

This PR adds an exception to take this scenario into account.

#### Any other comments?

**Edit:** This fix is not enough to solve the issue when using an estimator inside a pipeline, i.e. the following code raises the same `ValueError`:

```python
from sklearn.datasets import make_regression
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import make_scorer, mean_absolute_error
from sklearn.model_selection import cross_validate
from sklearn.multioutput import RegressorChain
from sklearn.pipeline import make_pipeline

X, Y = make_regression(n_features=10, n_targets=2, random_state=0)
hist_gbdt = HistGradientBoostingRegressor(random_state=0)
model = make_pipeline(RegressorChain(hist_gbdt))

scoring = {"MO_MAE": make_scorer(mean_absolute_error, multioutput="raw_values")}
cv_results = cross_validate(model, X, Y, scoring=scoring)
cv_results
```

I am a bit lost on how to deal with such case. Any comments are quite appreciated.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->
user: ArturoAmorQ
issue_25686: state: open
title: ⚠️ CI failed on Linux_Docker.debian_atlas_32bit ⚠️
body: **CI is still failing on [Linux_Docker.debian_atlas_32bit](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=61913&view=logs&j=aabdcdc3-bb64-5414-b357-ed024fe8659e)** (Dec 31, 2023)
- test_regression_dataset[9-0.5-absolute_error]
user: scikit-learn-bot
issue_25682: state: open
title: DOC Improve narrative of plot_cluster_comparison.py example
body: #### Reference Issues/PRs

Somewhat related to #22406


#### What does this implement/fix? Explain your changes.

This PR implements cosmetic changes to the cluster comparison example with the intention of making the contents more didactic.


#### Any other comments?

No.


user: cavorit
issue_25666: state: open
title: Multioutput regressors raise ValueError when scoring with `multioutput="raw_values"`
body: ### Describe the bug

The goal of the `multioutput="raw_values"` parameter in the regression metrics is to be able to inspect the individual scores of a multioutput metaestimator, but the `_score` function in `_validation.py` expects a number, not the array it actually outputs.

We should add an exception to take into account this scenario.

### Steps/Code to Reproduce

```python
from sklearn.datasets import make_regression
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import make_scorer, mean_absolute_error
from sklearn.model_selection import cross_validate
from sklearn.multioutput import MultiOutputRegressor

X, Y = make_regression(n_features=10, n_targets=2, random_state=0)
hist_gbdt = HistGradientBoostingRegressor(random_state=0)
model = MultiOutputRegressor(hist_gbdt) # RegressorChain fails as well
scoring = {"MO_MAE": make_scorer(mean_absolute_error, multioutput="raw_values")}

cv_results = cross_validate(model, X, Y, scoring=scoring)
cv_results["test_MO_MAE"]
```

### Expected Results

Array of shape (`n_outputs`, `n_splits`), in this case shape (2, 5) as using the default KFold cross-validation.

### Actual Results

```python-traceback
File ~/miniforge3/envs/joblib-benchmark/lib/python3.9/site-packages/sklearn/model_selection/_validation.py:708, in _fit_and_score(estimator, X, y, scorer, train, test, verbose, parameters, fit_params, return_train_score, return_parameters, return_n_test_samples, return_times, return_estimator, split_progress, candidate_progress, error_score)
    705 result["fit_error"] = None
    707 fit_time = time.time() - start_time
--> 708 test_scores = _score(estimator, X_test, y_test, scorer, error_score)
    709 score_time = time.time() - start_time - fit_time
    710 if return_train_score:

File ~/miniforge3/envs/joblib-benchmark/lib/python3.9/site-packages/sklearn/model_selection/_validation.py:809, in _score(estimator, X_test, y_test, scorer, error_score)
    807                 score = score.item()
    808         if not isinstance(score, numbers.Number):
--> 809             raise ValueError(error_msg % (score, type(score), name))
    810         scores[name] = score
    811 else:  # scalar

ValueError: scoring must return a number, got [110.4080722   83.27384784] (<class 'numpy.ndarray'>) instead. (scorer=MO_MAE)
```

### Versions

```shell
System:
    python: 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03)  [GCC 11.3.0]
executable: /home/arturoamor/miniforge3/envs/joblib-benchmark/bin/python3.9
   machine: Linux-5.14.0-1057-oem-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.2.1
          pip: 23.0.1
   setuptools: 67.3.2
        numpy: 1.24.2
        scipy: 1.10.1
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.7.0
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/arturoamor/miniforge3/envs/joblib-benchmark/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/arturoamor/miniforge3/envs/joblib-benchmark/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so
        version: 0.3.21
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/arturoamor/miniforge3/envs/joblib-benchmark/lib/python3.9/site-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 8
```

user: ArturoAmorQ
issue_25655: state: open
title: Implement class to reverse item encodings
body: ### Describe the workflow you want to enable

Hi! As a psychologist working with questionnaire data, you will find yourselves in situations where you have to reverse the values in certain variables (i.e. questionnaire items), because they have reversed logic (see [this link](https://psych.hanover.edu/classes/ResearchMethods/jamovi/reliability/reverseScoring.html) for an example).

### Describe your proposed solution

Implement a class that reverses variable values:

```python
class ReverseColumnValues(TransformerMixin,BaseEstimator):
    '''Reverse matrix column values. NaN values stay as they are'''
    
    def fit(self, X, y=None):
        return self
    
    def _reverse_array_values(self,a):
        '''Reverse values in a vector'''
        unique_vals = np.unique(a[~np.isnan(a)])  # Get unique non-nan values
        flip_dict = dict(zip(unique_vals,np.flip(unique_vals)))  # Create flip dictionary
        return np.vectorize(flip_dict.get)(a)  # map old values onto new values

    def _reverse_column_values(self,X):
        '''Reverse values of all columns of a matrix'''
        X_ = np.apply_along_axis(self._reverse_array_values,0,X)
        return X_
    
    def transform(self, X, y=None):
        return self._reverse_column_values(X)
```



### Describe alternatives you've considered, if relevant

Most certainly users would have to use this class as input to `ColumnTransformer` because you only want to reverse certain variables and leave the rest as it is.

### Additional context

Not sure though, if this might out of scope of scikit-learn. If not, maybe there's a contrib project that could use this?
user: JohannesWiesner
labels: Needs Investigation: Issue requires investigation
issue_25646: state: open
title: feat: Support sample weight for MLP
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
sample_weight support will contributes to class_weight support, so this PR contributes to the issue https://github.com/scikit-learn/scikit-learn/issues/9113, and the stalled PR https://github.com/scikit-learn/scikit-learn/pull/11723 

#### What does this implement/fix? Explain your changes.

- [x] Add sample_weight support

 

   - [x] Add sample_weight support for multiple loss functions:
     - [x] log_loss
     - [x] binary_log_loss
     - [x]  squared_loss
   - [x]  Add sample_weight parameter to fit and partial_fit
     - [x]  Propagate to relevant functions
     - [x]  Validate sample_weight for both MLPClassifier and MLPRegressor
     - [x]  Handle zero weights
   - [x]  Add tests
     - [x]  Test directly loss functions sample_weight invariance
     - [x]  Test fit validation of sample_weight
     - [x]  Test sample weight effect for both MLPClassifier and MLPRegressor
     - [x] Test sample weight together with early stopping


#### Any other comments? 
I have a draft PR https://github.com/scikit-learn/scikit-learn/pull/25326 to support class weight for MLP which will depend on this PR for sample weight support first. 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: zshu115x
issue_25645: state: open
title: DOC include note for searching for optimal parameters with successive halving
body: #### Reference Issues/PRs
Fixes #24634

#### What does this implement/fix? Explain your changes.
Includes a note for how resources should be increased.

#### Any other comments?
Would it be helpful to include a comment in the note that states that "For estimators such as RandomForest a plateau might be reached while increasing `n_estimators`"?
user: rprkh
issue_25642: state: open
title: SimpleImputer with the rule strategies median-1/median+1
body: ### Describe the workflow you want to enable

I have run into an issue with `SimpleImputer`. Given a feature of, say, integer type, it is completely reasonable to impute the median to missing values. However, when the overall number of records is even, there is a decent chance, that the median falls between two integers according to the well-known rule (Sorted[N/2-1] + Sorted[N/2])/2. The issue is, that technically, this kind of imputation breaks the domain of the feature, it used to be integer, but now there are spectacular .5 numbers, which can act weirdly in further processing.

Long story, short, when a sequence like 4, 3, ?, 2, 4, 5, 1 is imputed by 3.5, it is not an integer sequence anymore.

### Describe your proposed solution

My recommendation is to introduce something like an "adjusted median", which would ensure that the imputed value is a value of the domain of the feature. My recommendation is to pick Sorted[N/2-1] or Sorted[N/2], whichever has the highest number of occurances in the data. If equal, take the smallest.

Basically the "most_frequent" strategy applied to Sorted[N/2-1] and Sorted[N/2] only.

### Describe alternatives you've considered, if relevant

Alternative solutions and strategy names could work as well. In the problem described above, the issue is that median calculation is limited to its mathematical definition. `np.percentile`, just like percentile functions in `R` offer more flexibility, as what happens in `SimpleImputer` with the `strategy=median` is that the 50% percentile is taken with linear interpolation. `np.percentile` could do it with `nearest` interpolation. I think offering this control would improve the flexibility of the imputer with very little effort.

### Additional context

_No response_
user: gykovacs
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_25639: state: open
title: FEA Implementation of "threshold-dependent metric per threshold value" curve
body: Towards #21391.

Intending to later build the `MetricThresholdCurveDisplay` following the same structure that other Displays have, this PR implements the associate curve. I decided to break the original issue into two parts (curve and Display) for easier review (but I don't mind adding the Display to this PR as well).

A quick example of usage of the implementation here:
```python
import matplotlib.pyplot as plt
from imblearn.datasets import fetch_datasets
from sklearn.inspection import metric_threshold_curve
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import fbeta_score
from functools import partial

dataset = fetch_datasets()["coil_2000"]
X, y = dataset.data, (dataset.target==1).astype(int)

X_train_model, X_test, y_train_model, y_test = train_test_split(X, y, random_state=0, stratify=y)

model = RandomForestClassifier(random_state=0).fit(X_train_model, y_train_model)
predict_proba = model.predict_proba(X_test)[:, 1]

f2_values, thresholds = metric_threshold_curve(
    y_test, predict_proba, partial(fbeta_score, beta=2), threshold_grid=500
)

fig, ax = plt.subplots(figsize=(5, 2.4))
ax.plot(thresholds, f2_values)
ax.set_xlabel("thresholds")
ax.set_ylabel("f2 score")
plt.tight_layout()
```
![image](https://user-images.githubusercontent.com/55899543/219841134-c0cb37cc-2b35-4312-a07a-b128f3a132ee.png)

Most of the code for `metric_threshold_curve` function is an adaptation of [`_binary_clf_curve`](https://github.com/scikit-learn/scikit-learn/blob/22336afec6384e85366d94dab7a108fd2a0b64fc/sklearn/metrics/_ranking.py#L732).

Points of doubt:
- I thought the **inspection** module would be suitable for this type of analysis, but it is not 100% clear to me that this curve (and then the Display) should go here - other current options would be **metrics** or **model_selection._prediction** (just like the related meta-estimator from #16525).
- I would appreciate some help with **test** ideas!
- I made the first version of the documentation to go along with the function. It's preliminary. I wanted to go into it only a little while we don't define how the function will look like. Ideas for it would be appreciated as well! :)
user: vitaliset
issue_25631: state: open
title: Inflated results on random-data with SVM
body: ### Describe the bug

When trying to train/evaluate a support vector machine in scikit-learn, I am experiencing some unexpected behaviour and I am wondering whether I am doing something wrong or that this is a possible bug.

In a very specific subset of circumstances, namely:

- `LeaveOneOut() `is used as cross-validation procedure
- The SVM is used, with `probability = True` and a small `C`  such as` 0.01`
- The y labels are balanced (i.e. the mean of y is 0.5)

The results of the trained SVM are very good on randomly generated data - while they should be near chance. If the y labels are a bit different, or the SVM is swapped out for a ``LogisticRegression``, it gives expected results (Brier of 0.25, AUC near 0.5). 
But for the named circumstances, the Brier is roughly 0.10 - 0.15 and AUC > 0.9 if the y labels are balanced.

### Steps/Code to Reproduce
```python

from sklearn import svm
from sklearn.linear_model import LogisticRegression
import numpy as np
from sklearn.model_selection import GridSearchCV, StratifiedKFold, LeaveOneOut, KFold
from sklearn.metrics import roc_auc_score, brier_score_loss
from tqdm import tqdm
import pandas as pd


N = 20
N_FEATURES = 50


scores = []
for z in tqdm(range(500)):
    X = np.random.normal(0, 1, size=(N, N_FEATURES))
    y = np.random.binomial(1, 0.5, size=N)
    
    if z < 10:
        y = np.array([0, 1] * int(N/2))
        y = np.random.permutation(y)

    y_real, y_pred = [], []
    skf_outer = LeaveOneOut()
    for train_index, test_index in skf_outer.split(X, y):
        X_train, X_test = X[train_index], X[test_index, :]
        y_train, y_test = y[train_index], y[test_index]

        clf = svm.SVC(probability=True, C=0.01)

        clf.fit(X_train, y_train)
        predictions = clf.predict_proba(X_test)[:, 1]

        y_pred.extend(predictions)
        y_real.extend(y_test)

    scores.append([np.mean(y), 
                   brier_score_loss(np.array(y_real), np.array(y_pred)), 
                   roc_auc_score(np.array(y_real), np.array(y_pred))])

df_scores = pd.DataFrame(scores)
df_scores.columns = ['y_label', 'brier', 'auc']
df_scores['y_0.5'] = df_scores['y_label'] == 0.5
df_scores = df_scores.groupby(['y_0.5']).mean()
print(df_scores)
```

### Expected Results

I would expect that all results would be somewhat similar, with a Brier ~0.25 and AUC ~0.5.

### Actual Results

```
        y_label     brier       auc
y_0.5                              
False  0.514649  0.298204  0.216884
True   0.500000  0.159728  0.999080
```

Here, you can see that if the ``np.mean`` of the ``y_labels`` is 0.5, the results are actually really really good.
While the data is randomly generated for 500 times

### Versions

```shell
System:
    python: 3.8.15 (default, Nov 24 2022, 14:38:14) [MSC v.1916 64 bit (AMD64)]
executable: C:\ProgramData\Anaconda3\envs\test\python.exe
   machine: Windows-10-10.0.19044-SP0
Python dependencies:
      sklearn: 1.2.0
          pip: 22.2.2
   setuptools: 61.2.0
        numpy: 1.19.5
        scipy: 1.10.0
       Cython: 0.29.14
       pandas: 1.4.4
   matplotlib: 3.6.3
       joblib: 1.2.0
threadpoolctl: 2.2.0
Built with OpenMP: True
threadpoolctl info:
       filepath: C:\ProgramData\Anaconda3\envs\test\Library\bin\mkl_rt.1.dll
         prefix: mkl_rt
       user_api: blas
   internal_api: mkl
        version: 2021.4-Product
    num_threads: 8
threading_layer: intel
       filepath: C:\Users\manuser\AppData\Roaming\Python\Python38\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
         prefix: libopenblas
       user_api: blas
   internal_api: openblas
        version: 0.3.18
    num_threads: 16
threading_layer: pthreads
   architecture: Prescott
       filepath: C:\ProgramData\Anaconda3\envs\test\Lib\site-packages\sklearn\.libs\vcomp140.dll
         prefix: vcomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 8
       filepath: C:\ProgramData\Anaconda3\envs\test\Library\bin\libiomp5md.dll
         prefix: libiomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 8
       filepath: C:\Users\manuser\AppData\Roaming\Python\Python38\site-packages\mxnet\libopenblas.dll
         prefix: libopenblas
       user_api: blas
   internal_api: openblas
        version: None
    num_threads: 16
threading_layer: pthreads
   architecture: Prescott
       filepath: C:\ProgramData\Anaconda3\envs\test\Lib\site-packages\torch\lib\libiomp5md.dll
         prefix: libiomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 16
       filepath: C:\ProgramData\Anaconda3\envs\test\Lib\site-packages\torch\lib\libiompstubs5md.dll
         prefix: libiomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 1
```

user: CriticalValue
labels: Needs Investigation: Issue requires investigation
issue_25623: state: open
title: KernelDensity incorrect handling of bandwidth
body: ### Describe the bug

I was using kernel density estimator
https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KernelDensity.html
using 'silverman' or 'scott' as the bandwidth argument. Then I found that the bandwidth automatically adjusted by the algorithm is independent of the actual scale of the dataset. In fact, I was shocked to find that the calculation of a bandwidth in https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neighbors/_kde.py for 'silverman' and 'scott' does not check the scales of data at all. 

Suppose I fit the model `kde` to some 2D data `X` and get the bandwidth as `kde.bandwidth_`.
Next, I fit the model `kde` to the same 2D data `X` but with all elements multiplied by, say, 20 and get the bandwidth as `kde.bandwidth_`. 
I found that these two values of `kde.bandwidth_` are equal (it is calculated from the shape of `X`, see the source code). But obviously they should differ by a factor of 20 if the bandwidth is really computed in a truly adaptive manner.

For your reference, I want to mention that scipy's KDE https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html calculates the covariance of data to extract the scale of data. I think this is the right thing to do.

Note that if the bandwidth is incorrect, everything else is incorrect too, including probablities of samples, etc.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.neighbors import KernelDensity
X = np.random.randn(1000, 2)
kde = KernelDensity(bandwidth='scott')
kde.fit(X)
print(kde.bandwidth_)

kde.fit(X * 20)
print(kde.bandwidth_)
```

### Expected Results

Different bandwidths for data sets with different scales.

### Actual Results

0.31622776601683794
0.31622776601683794

### Versions

```shell
1.2.1
```

user: TaTKSM
issue_25617: state: open
title: FEA Add a private `check_array` with additional parameters
body: To allow plugins or other users to pass in a custom `asarray` function an additional parameter is added to a private version of `check_array`. If the caller passes something, that callable is used instead of the default `np.asarray`. This is useful if you want to directly convert to a, say, cupy array. Without this parameter first a numpy array would be created which you then convert to a cupy array. This also makes it possible for cupy arrays to be passed in to `_validate_data`. This is useful for plugin authors who want to re-use the validation methods of scikit-learn, instead of having to maintain a copy of them.

The reason for the indirection is to avoid increasing the number of public parameters that need to then follow the deprecation policy.


#### Reference Issues/PRs
This is an attempt to closes #25433

#### What does this implement/fix? Explain your changes.
This implements the "add a private version with this additional parameter" idea from #25433.
user: betatim
issue_25612: state: open
title: Simplify decision tree removing redundant decisions
body: ### Describe the workflow you want to enable

Description: Add a new method simplify() to the decision tree Class that returns a simplified version of the decision tree by pruning redundant leaves that do not add new decision paths. This simplification method will create a new instance of the tree with fewer nodes, allowing users to obtain a more concise and interpretable tree.

Motivation: Decision trees are known for their ability to provide easily interpretable models, but sometimes they can grow to a size that makes interpretation challenging. A simplified version of the decision tree will make it easier for users to understand the model, especially in situations where the tree has a large number of redundant leaves.

Example:
![image](https://user-images.githubusercontent.com/11466701/218794306-eac085c6-2426-4e1f-aaf9-d563b2561922.png)
Can be simplified to
![image](https://user-images.githubusercontent.com/11466701/218795262-0a625e63-633d-44e0-97b2-653b0b8fc27a.png)

Related Work: There are related issues in [Scikit-learn #10810](https://github.com/scikit-learn/scikit-learn/issues/10810), [IBM Taxinomitis #226](https://github.com/IBM/taxinomitis/issues/226) and [Stack Overfflow](https://stackoverflow.com/questions/51397109/prune-unnecessary-leaves-in-sklearn-decisiontreeclassifier) that point to the need for decision tree simplification. Scikit-learn has added post-pruning methods based on cost, as far as I know, they do not achieve the same level of simplification proposed in this feature.



### Describe your proposed solution

Proposed Solution: Add a simplify() method to the decision tree module that returns a simplified version of the tree. The method will prune redundant leaves that do not add new decision paths. The new tree will have fewer nodes and be easier to interpret.

Example Usage:
```Python
from sklearn.tree import DecisionTreeClassifier

clf = DecisionTreeClassifier()

...

clf.fit(X, y)

tree = clf.tree_
simplified_tree = tree.simplify()
```


### Describe alternatives you've considered, if relevant

An alternative way is to create ad hoc functions to achieve this simplification. But I believe this feature is important enough to be in the main repository

### Additional context

I can work on this feature request and send a PR, if approved by members.
user: nullhack
labels: Needs Investigation: Issue requires investigation
issue_25605: state: open
title: MNT [engine-api] revert NotSupportedByEngineError feature and automatic catch in unit tests
body: #### What does this implement/fix? Explain your changes.

Follow-up on latest discussions regarding the plugin api this PR remove the `NotSupportedByEngineError` exception and its usage in unit tests.

user: fcharras
issue_25604: state: open
title: MLPR with solver='lbfgs', nonzero alpha doesn't make the same result from multiple run.
body: ### Describe the bug

I have tested the simple regression modeling with MLPRegressor(solver='lbfgs', alpha=0.01, tol=0.0001, random_state=42) with sample data with three input parameters and three targets.
I tested multiple run in Linux OS. But, sometimes I got different regression.score(X_test, y_test).
But, if I change the solver to 'adam' or set alpha to zero, it makes the same value all the time.
So, I am suspicious that there is randomness with respect to solver, alpha, and tol. But, it's hard to find the cause from the source code.
I wonder if anybody has the same experience.

### Steps/Code to Reproduce

```py
import warnings
warnings.simplefilter('ignore')


for k in range(100):

    import numpy as np
    import pandas as pd
    from sklearn.model_selection import train_test_split
    from sklearn.preprocessing import StandardScaler
    from sklearn.neural_network import MLPRegressor

    data = np.array(
            [
                [1.3436424, 8.4743374, 5.0, 67.34851990996208, 130.0, 126.0],
                [1.3436424, 8.4743374, 1.0, 63.34851990996208, 130.0, 126.0],
                [7.6377462, 2.5506903, 5.0, 277.4131267778526, 130.0, 126.0],
                [7.6377462, 2.5506903, 1.0, 273.4131267778526, 130.0, 126.0],
                [4.9543509, 4.4949106, 5.0, 189.98360040364014, 130.0, 126.0],
                [4.9543509, 4.4949106, 1.0, 85.98360040364014, 130.0, 126.0],
                [6.5159297, 7.8872335, 5.0, 389.335314633451, 130.0, 126.0],
                [6.5159297, 7.8872335, 1.0, 385.335314633451, 130.0, 126.0],
                [0.93859587, 0.28347477, 5.0, 8.97322786339917, 130.0, 126.0],
                [0.93859587, 0.28347477, 1.0, 4.9732278633991704, 130.0, 126.0],
                [8.357651, 4.3276707, 5.0, 395.3967974810315, 130.0, 126.0],
                [8.357651, 4.3276707, 1.0, 391.3967974810315, 130.0, 126.0],
                [7.6228008, 0.021060534, 5.0, 180.12397738656006, 130.0, 126.0],
                [7.6228008, 0.021060534, 1.0, 176.12397738656006, 130.0, 126.0],
                [4.4538719, 7.2154003, 5.0, 225.1932679219367, 130.0, 126.0],
                [4.4538719, 7.2154003, 1.0, 221.1932679219367, 130.0, 126.0],
                [2.2876222, 9.452707, 5.0, 28.8207579062755, 130.0, 126.0],
                [2.2876222, 9.452707, 1.0, 124.82075790627552, 130.0, 126.0],
                [9.0142746, 0.30589983, 5.0, 262.5587650311821, 130.0, 126.0],
                [9.0142746, 0.30589983, 1.0, 258.5587650311821, 130.0, 126.0],
                [0.25445861, 5.4141247, 5.0, 12.082600780252731, 130.0, 126.0],
                [0.25445861, 5.4141247, 1.0, 8.082600780252731, 130.0, 126.0],
                [9.3914916, 3.8120424, 5.0, 448.60416431083087, 130.0, 126.0],
                [9.3914916, 3.8120424, 1.0, 444.60416431083087, 130.0, 126.0],
                [2.165994, 4.2211658, 5.0, 64.78968900313399, 130.0, 126.0],
            ],
            dtype=object,
        )

    df = pd.DataFrame(
        data, columns=["a", "b", "c", "t0", "t1", "t2"]
    )

    X = df[["a", "b", "c"]]
    y = df[["t0", "t1", "t2"]]

    scaler_x = StandardScaler()
    X = scaler_x.fit_transform(X)
    scaler_y = StandardScaler()
    y = scaler_y.fit_transform(y)

    random_state = 42

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=random_state)

    reg = MLPRegressor(max_iter=200, solver='lbfgs', alpha=0.01, tol=0.0001, random_state=42)
    reg.fit(X_train, y_train)
    scores = reg.score(X_train, y_train), reg.score(X_test, y_test)
    print(k, scores[0], scores[1])
```

### Expected Results

0 0.33326240015324454 0.30623113513186795
1 0.33326240015324454 0.30623113513186795
2 0.33326240015324454 0.30623113513186795
3 0.33326240015324454 0.30623113513186795
4 0.33326240015324454 0.30623113513186795
5 0.33326240015324454 0.30623113513186795
...
98 0.33326240015324454 0.30623113513186795
99 0.33326240015324454 0.30623113513186795

### Actual Results

It makes two different output values randomly.

### Versions

```shell
1.1.3
```

user: yeongkwoncho
issue_25602: state: open
title: DOC improve EDA for `plot_cyclical_feature_engineering.py`
body: #### Reference Issues/PRs
No issue. Based on the discussion in https://github.com/scikit-learn/scikit-learn/pull/25350#discussion_r1100267964

#### What does this implement/fix? Explain your changes.
Includes the following plots:
- Hourly count of bikes over different days of the week
- Hourly count of bikes over different seasons

#### Any other comments?
Improves the EDA for the example.

/cc @ArturoAmorQ 
user: rprkh
issue_25594: state: open
title: KBinsDiscretizer creates wrong bins.
body: ### Describe the bug

`KBinsDiscretizer` gives wrong bins and wrong transformed data, when the inputs contains only 2 distinct values, and `n_bins=3`. It only produces 1 bin, which is not expected.  The warning shows some bins are too small so they are merged.

Note that when setting `n_bins=2`, `KBinsDiscretizer` gives more reasonable bins and correct results. However, from the `bin_edges_`, it seems `n_bins=2` is handled separately.

### My guess about the cause
I think the root cause is inconsistent handling of `-inf` and `+inf` in fitting and transforming. 

In fitting, `-inf` and `+inf` are not considered when removing duplicated bin edges. But in transforming data, the first and last bin edges are replaces with `-inf` and `+inf`. Such differences obviously removes some necessary bins.

Taking the case I mentioned as a example, current implementation deduplicates bin edges from `[-1.0, -1.0, 0.0, 0.0]` to `[-1.0, 0.0]`.  Then in transforming, all real values are actually in one bin, since the bin edges are replaced with `-inf` and `+inf`.

A more reasonable conversion is replacing first and last bin edges with `-inf` and `+inf` in fitting, i.e., `[-inf, -1.0, 0.0, inf]`, then removing duplicated edges, which gives `[-inf, -1.0, 0.0, inf]`. This will give expected results.




### Steps/Code to Reproduce

```
from sklearn.preprocessing import KBinsDiscretizer
X = [[-1], [-1], [ 0], [ 0]]
discretizer = KBinsDiscretizer(n_bins=3, encode='ordinal', strategy='quantile')
discretizer.fit(X)

print(discretizer.bin_edges_)
Xt = discretizer.transform(X)
print(Xt)
```

### Expected Results

The last `print` should give 
```
[[0.]
 [0.]
 [1.]
 [1.]]
```
or 

```
[[1.]
 [1.]
 [2.]
 [2.]]
```
. It depends on how to implement it correctly.

### Actual Results

The last `print` gives
```
[[0.]
 [0.]
 [0.]
 [0.]]
```

### Versions

```shell
System:
    python: 3.8.16 (default, Jan 17 2023, 16:42:09)  [Clang 14.0.6 ]
executable: /opt/anaconda3/envs/sklearn/bin/python3
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.2.1
          pip: 22.3.1
   setuptools: 65.6.3
        numpy: 1.24.2
        scipy: 1.10.0
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /opt/anaconda3/envs/sklearn/lib/python3.8/site-packages/sklearn/.dylibs/libomp.dylib
        version: None
    num_threads: 10

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/anaconda3/envs/sklearn/lib/python3.8/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.21
threading_layer: pthreads
   architecture: Nehalem
    num_threads: 10

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/anaconda3/envs/sklearn/lib/python3.8/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.18
threading_layer: pthreads
   architecture: Nehalem
    num_threads: 10
```
```

user: Fanoid
labels: Needs Decision: Requires decision
issue_25592: state: open
title: set_config(transform_output="pandas") does not act on inverse_transform
body: ### Describe the bug

The new [`set_config(transform_output='pandas')` functionality](https://blog.scikit-learn.org/technical/pandas-dataframe-output-for-sklearn-transformer/) is very useful, but unfortunately it is only taking an effect on the `transform` method of a transformer and not on the `inverse_transform`.

### Steps/Code to Reproduce

```python
import pandas as pd
from sklearn import set_config
from sklearn.datasets import load_diabetes
from sklearn.preprocessing import StandardScaler

set_config(transform_output="pandas")

# Load the diabetes dataset
X, _ = load_diabetes(return_X_y=True)
X = pd.DataFrame(X)
scaler = StandardScaler()
Xt = scaler.fit_transform(X)
assert isinstance(Xt, pd.DataFrame)  # OK!

X = scaler.inverse_transform(Xt)
assert isinstance(X, pd.DataFrame)  # AssertionError!
```

### Expected Results

Output of `StandardScale.inverse_transform()` to be a pandas DataFrame.

### Actual Results

---------------------------------------------------------------------------                                                                                 
AssertionError                            Traceback (most recent call last)
Cell In[25], line 16
     13 assert isinstance(Xt, pd.DataFrame)
     15 X = scaler.inverse_transform(Xt)
---> 16 assert isinstance(X, pd.DataFrame)

AssertionError: 

### Versions

```shell
System:
    python: 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]
executable: /usr/bin/python3.10
   machine: Linux-5.14.0-1038-oem-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.2.1
          pip: 23.0
   setuptools: 65.3.0
        numpy: 1.22.4
        scipy: 1.9.0
       Cython: 0.29.30
       pandas: 1.5.1
   matplotlib: 3.5.3
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/afr/.local/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Prescott
    num_threads: 16

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/afr/.local/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-9f9f5dbc.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Prescott
    num_threads: 16

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
        version: None
    num_threads: 16

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/afr/.local/lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 16

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/afr/.local/lib/python3.10/site-packages/torch/lib/libgomp-a34b3233.so.1
        version: None
    num_threads: 8
```

user: afrendeiro
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_25590: state: open
title: Importing BaseEstimator leads to unnecessary memory usage
body: ### Describe the bug

Importing `BaseEstimator` from `sklearn.base` causes a cascade of imports that leads to unnecessary memory usage (500MiB of stuff at peak, see screenshot below).

![Screenshot from 2023-02-10 19-23-53](https://user-images.githubusercontent.com/1296726/218169206-d8e940e9-40ba-458d-8047-deacf2f30521.png)


### Steps/Code to Reproduce

```python
from sklearn.base import BaseEstimator
```

### Expected Results

I would expect such a basic class to import some utilities around, but nothing major.

### Actual Results

The code ends up importing a large amount of unnecessary and large modules, the full chain to some point is:
 - code -> `from sklearn.base import BaseEstimator`
 - sklearn/\_\_init\_\_.py:82 -> `from .base import clone`
 - sklearn/base.py:17 -> `from .utils import _IS_32BIT`
 - sklearn/utils/\_\_init\_\_.py:21 -> `from scipy.sparse import issparse` (This is the most useless import, as the issparse function is a oneliner that calls isinstance)
 - scipy/sparse/\_\_init\_\_.py:283 -> `from . import csgraph`
 - scipy/sparse/csgraph/\_\_init\_\_.py:185 -> `from ._laplacian import laplacian`
 - scipy/sparse/csgraph/_laplacian.py:7 -> `from scipy.sparse.linalg import LinearOperator`
 - scipy/sparse/linalg/\_\_init\_\_.py:120 -> `from ._isolve import *`
 - scipy/sparse/linalg/_isolve/\_\_init\_\_.py:4 -> `from .iterative import *`
 - scipy/sparse/linalg/_isolve/iterative.py:9 -> `from . import _iterative`

I guess the issue I am reporting is that importing a simple estimator base class should not cause so many unrelated imports and spike memory usage by doing so.

### Versions

```shell
System:
    python: 3.10.8 (main, Nov  1 2022, 14:18:21) [GCC 12.2.0]
executable: /home/jjan/dev/sec-certs-page/virt/bin/python
   machine: Linux-6.0.12-arch1-1-x86_64-with-glibc2.36

Python dependencies:
      sklearn: 1.2.1
          pip: 23.0
   setuptools: 61.2.0
        numpy: 1.22.3
        scipy: 1.10.0
       Cython: None
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/jjan/dev/sec-certs-page/virt/lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 16

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/jjan/dev/sec-certs-page/virt/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Zen
    num_threads: 16

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/jjan/dev/sec-certs-page/virt/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Zen
    num_threads: 16
```

user: J08nY
labels: Needs Investigation: Issue requires investigation
issue_25572: state: open
title: RFC Guideline for usage of Cython types
body: ## Goal
Have a documented consensus on which types to use in Cython code.

### Types
We should distinguish between floating point numbers and integers. We may also split the use cases of integers: As data value and as index for pointers and memoryviews.

## Linked issues
#24153, https://github.com/scikit-learn/scikit-learn/pull/25555#discussion_r1100738562, https://github.com/scikit-learn/scikit-learn/pull/23865#discussion_r926716129, https://github.com/scikit-learn/scikit-learn/pull/23865#discussion_r941346271

Also note the Cython bug with `const` types: https://github.com/cython/cython/issues/5230

user: lorentzenchr
issue_25565: state: open
title: High level documentation of the CI infrastructure
body: As originally discussed in https://github.com/scikit-learn/scikit-learn/pull/25562#discussion_r1098396646:

I think it might be helpful to give a high level description of our CI somewhere in the doc, both for new contributors and maintainers. In particular, we should summarize the various missions of our CI:

- building and testing PRs to give feedback to contributors and reviewers prior to merging (triggered by commits pushed to the PR branch on the contributor's fork),
- building the stable and dev doc websites (triggered by commits pushed to the `main` and `X.Y.Z` branches on the main repo respectively),
- building and testing scikit-learn against the development versions of its dependencies (triggered as scheduled jobs),
- building, testing, and publishing nightly wheels against stable dependencies (triggered as scheduled jobs),
- building, testing, and publishing release wheels to pypi.org (triggered by commits pushed to the `X.Y.Z` branch on the main repo + manually triggered upload github action workflow),

Ideally this doc should only describe the high level objectives of the CI infrastructure and point the readers to the config files and the `build_tools` folder for the details (which typically evolve faster than we update the doc).

We should carefully use inline comments in those config files and scripts to make sure that people easily understand how they operate and how to conduct common maintenance operations (e.g. rotating secret tokens).

This doc should also consolidate the list of commit message trigger keywords (currently present in the contributors guide if I am not mistaken).

I think the CI doc deserves its own page in our usage guide and should be cross-linked from the contributors and the maintainers docs.

user: ogrisel
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_25564: state: open
title: Streamlining Bug Fix Releases
body: Reading over https://github.com/scikit-learn/scikit-learn/pull/25457 I wish we had workflow where we can immediately backport fixes to `1.2.X` once the fix is on `main`. This way we do not need to do a big interactive rebase when we release. We would only need to update the authors list and then cut a release directly from `1.2.X`.

I think the only thing stopping us from directly back-porting fixes to `upstream/1.2.X` is that it updates the documentation right away. In that case, I see two ways around this:

1. A new `doc/1.2.X` branch that gets deployed to the docs. `doc/1.2.X` will always be behind `1.2.X`. When we release, sync `doc/1.2.X` up with `1.2.X`.
2. A new `dev/1.2.X` that we backport fixes to. `dev/1.2.X` will always be ahead of `1.2.X`. When we release, sync `1.2.X` up with `dev/1.2.X`.

I know this adds a little more process, but I think making it easier to cut bug fix releases.
user: thomasjpfan
labels: workflow: Development workflow changes
issue_25560: state: open
title: set_output API do not preserve original dtypes for pandas
body: ### Describe the bug

Following issue #24182,

When using the set_output with expected output to be a pandas' data frame, while converting tougher columns with different dtypes the output does not preserve the original dtype but the "common type" by numpy.

Possible workaround is to create transformer for each column, but it doesn't feel like the right approach.

In collaboration with @BenEfrati

### Steps/Code to Reproduce

```python
import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer



X = pd.DataFrame({
                  "age": [1, 2, 4],
                  "float": [1.5, 2.5, None],
                  })
X["age"] = X["age"].astype(np.uint8)
X.dtypes

# age        uint8
# float    float64
# dtype: object

simple_imputer = SimpleImputer(strategy='constant', fill_value=1).set_output(transform="pandas")
X_trans_partial = simple_imputer.fit_transform(X)
X_trans_partial.dtypes

# age      float64
# float    float64
# dtype: object
```

### Expected Results

We'd expect the age column to keep being an unsigned 8-bit integer.

### Actual Results

the age column is now a 64-bit floating point.

### Versions

```shell
System:
    python: 3.9.12 (main, Apr  4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)]
executable: C:\New folder\venv\Scripts\python.exe
   machine: Windows-10-10.0.19044-SP0

Python dependencies:
      sklearn: 1.2.1
          pip: 23.0
   setuptools: 58.1.0
        numpy: 1.24.2
        scipy: 1.10.0
       Cython: None
       pandas: 1.5.3
   matplotlib: None
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\New folder\venv\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
        version: 0.3.21
threading_layer: pthreads
   architecture: Haswell
    num_threads: 12

       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: C:\New folder\venv\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 12

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\New folder\venv\Lib\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell
    num_threads: 12
```

user: idanmoradarthas
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_25552: state: open
title: Implement beta calibration
body: ### Describe the workflow you want to enable

It would be nice to implement beta calibration as an additional option in CalibratedClassifierCV.

### Describe your proposed solution

Use the implementation provided in https://github.com/betacal/python (MIT license).

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Beta calibration seems like a generally superior alternative to logistic / Platt calibration. See here for further details, paper references, etc.: https://betacal.github.io/ It was proposed in 2017 in two papers by Meelis Kull, Telmo de Menezes e Silva Filho and Peter Flach.
user: e-pet
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_25535: state: open
title: [DRAFT] Engine plugin API and engine entry point for Lloyd's KMeans
body: This is a draft pull-request to allow third-party packages such as https://github.com/soda-inria/sklearn-numba-dpex to contribute alternative implementations of core computational routines of CPU-intensive scikit-learn estimators.

This would be particularly useful to experiment with GPU-optimized alternatives to our CPU-optimized Cython code.

This Draft PR serves as a long running design experiment (feature branch) to tackle #22438.

This is a takeover of the old PR (#24497) from the `ogrisel/scikit-learn:wip-engines` branch so as to make it easier to get the CI work when doing sub PRs targeting `scikit-learn/scikit-learn:feature/engine-api`.
user: ogrisel
issue_25516: state: open
title: Post-processing predicted probabilities using a helper (ie. exogenous) feature
body: ### Describe the workflow you want to enable

Here's the thing, I'm building a machine learning pipeline using Pipeline

Because I need to correct the prediction probability according to the value of a certain column after the classifier predicts.

For example, I have 12 columns of data, I need to use the first 11 columns for prediction, and then correct the prediction result according to the value of the 12th column (if the twelfth column is greater than 0, correct the prediction result to 0)

So my code is written as follows
```python
def modify_proba(X):
    X[:, 11][X[:, 11] > 0] = 0
    return X


model_1 = Pipeline(
    steps=[
        ("mapper", mapper),
        ("classifier", clf_1),
        ('modify_proba', FunctionTransformer(modify_proba, validate=False))])
```
But it prompts the following error


```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-193-399ece4a3431> in <module>
      3         ("mapper", mapper),
      4         ("classifier", clf_1),
----> 5         ('modify_proba', FunctionTransformer(modify_proba, validate=False))])

~/.local/lib/python3.7/site-packages/sklearn2pmml/pipeline/__init__.py in __init__(self, steps, header, predict_transformer, predict_proba_transformer, apply_transformer, memory, verbose)
     54                 self.apply_transformer = apply_transformer
     55                 # SkLearn 0.24+
---> 56                 super(PMMLPipeline, self).__init__(steps = steps, memory = memory, verbose = verbose)
     57 
     58         def __repr__(self):

~/.local/lib/python3.7/site-packages/sklearn/pipeline.py in __init__(self, steps, memory, verbose)
    132         self.memory = memory
    133         self.verbose = verbose
--> 134         self._validate_steps()
    135 
    136     def get_params(self, deep=True):

~/.local/lib/python3.7/site-packages/sklearn/pipeline.py in _validate_steps(self)
    180                                 "transformers and implement fit and transform "
    181                                 "or be the string 'passthrough' "
--> 182                                 "'%s' (type %s) doesn't" % (t, type(t)))
    183 
    184         # We allow last estimator to be None as an identity transformation

TypeError: All intermediate steps should be transformers and implement fit and transform or be the string 'passthrough' 'LogisticRegression(C=1, class_weight=None, dual=False, fit_intercept=True,
                   intercept_scaling=1, l1_ratio=None, max_iter=4000,
                   multi_class='ovr', n_jobs=16, penalty='l2', random_state=0,
                   solver='lbfgs', tol=0.0001, verbose=3, warm_start=False)' (type <class 'sklearn.linear_model._logistic.LogisticRegression'>) doesn't
```

It seems that adding an extra step after the classifier is not allowed?

How can I implement adding extra rules after the classifier? Because I noticed that the result after the classifier seems to be an array
![](https://pic.liuzaoqi.com/picgo/202301311417546.png)


I expect to use the values of certain columns to modify the probability that the prediction is 0, 1. Is it impossible to achieve?

### Describe your proposed solution

Post-processing predicted probabilities using a helper (ie. exogenous) feature

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: liuhuanshuo
issue_25492: state: open
title: Enable feature selectors to pass pandas DataFrame to estimator
body: ### Describe the workflow you want to enable

When running SequentialFeatureSelector (or, presumably, other feature selection methods) with a pandas DataFrame input, the reduced-feature input is passed to the estimator as a numpy array. This seems inconsistent with the other transformers that successfully maintain the pandas indices.

My actual use case is the following: Select the best k features that complement a set of fixed features. I was going to implement it like this: (happy to take suggestions for alternatives).

```python
import pandas as pd
import numpy as np
import sklearn
from sklearn.feature_selection import SequentialFeatureSelector
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import FunctionTransformer
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline

X=pd.DataFrame(np.random.rand(10,3))
y=pd.Series(np.random.rand(10))

fixed_features=pd.DataFrame(np.random.rand(X.shape[0],2))
appender= FunctionTransformer(lambda X:pd.concat([X,fixed_features.reindex(X.index)],axis=1))
X.shape,appender.fit_transform(X).shape
```

    ((10, 3), (10, 5))


```python
LR_with_fixed=LR=Pipeline(steps=[
    ('append',appender),
    ('LR',LinearRegression())
])
SFS_with_fixed = SequentialFeatureSelector(
    LR_with_fixed,n_features_to_select=1, direction="forward"
)
_=SFS_with_fixed.fit(X,y)
```

    ValueError: 
    All the 5 fits failed.
    .....
      File "/tmp/ipykernel_275/897617596.py", line 2, in <lambda>
        appender= FunctionTransformer(lambda X:pd.concat([X,fixed_features.reindex(X.index)],axis=1))
    AttributeError: 'numpy.ndarray' object has no attribute 'index'



### Describe your proposed solution

I believe that it should be possible to pass the reduced matrix to the estimator as a pandas DataFrame, and this would be consistent with the way other Transformers work currently.

### Describe alternatives you've considered, if relevant

An alternative that would suffice for my particular use case is to directly pass a set of fixed features as a parameter to the feature selector.

### Additional context

_No response_
user: chubukov
issue_25462: state: open
title: FEA add newton-lsmr solver to LogisticRegression and GLMs
body: #### Reference Issues/PRs
Supersedes #23507.
Fixes #16634.

#### What does this implement/fix? Explain your changes.
This PR adds a further `NewtonSolver`: `NewtonLSMRSolver`.
This solver uses the iteratively reweighted least squares (IRLS) formulation of a Newton step. This means the inner solver uses the square root of the Hessian and solves the corresponding least squares problem (as opposed to solving the normal equation as `"newton-cholesky"` is doing) with the iterative [LSMR](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.lsmr.html) solver.

This solver is therefore suited for dense and sparse `X`.

#### Any other comments?
The multinomial/multiclass case deserves special attention as there are different ways to look at the Hessian $X' W X$:
- Naive: Use, e.g. for `n_classes=3`
  ```
  X = [X 0 0]  W = [W00 W10 W00]  y = [y==0    0    0]  coef = [coef_class_0]  ...
      [0 X 0]      [W10 W11 W00]      [   0 y==1    0]         [coef_class_1]
      [0 0 X]      [W20 W10 W22]      [   0    0 y==2]         [coef_class_2]
  ```
- Consider every 1d-array/2d-array as a 2d-/3d-array with its 2nd/3rd dimension having all `n_samples`. Then
  $W = \mathrm{diag}(p) - p'p$ and $p$ the probability array in n_classes (and n_samples as "depth").
  Now, one can use the LDL decomposition of this particular matrix $W$, analytically given in https://doi.org/10.1111/j.2517-6161.1992.tb01875.x, use $\sqrt{D} L'$ as square root of $W$.
  This is the chosen approach.

user: lorentzenchr
issue_25460: state: open
title: HistGradientBoosting avoid data shuffling when early_stopping activated
body: hello, it would be useful if the _HistGradientBoostingRegressor_ or _HistGradientBoostingClassifier_ model had the ability to avoid data shuffling when using the _early_stopping_ and _validation_fraction_ parameters, since maintaining data order is a basic requirement in case you work with TimeSeries

  https://github.com/scikit-learn/scikit-learn/blob/98cf537f5/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py#L427
```python
            if sample_weight is None:
                X_train, X_val, y_train, y_val = train_test_split(
                    X,
                    y,
                    test_size=self.validation_fraction,
                    stratify=stratify,
                    random_state=self._random_seed,
                )
                sample_weight_train = sample_weight_val = None
            else:
                # TODO: incorporate sample_weight in sampling here, as well as
                # stratify
                (
                    X_train,
                    X_val,
                    y_train,
                    y_val,
                    sample_weight_train,
                    sample_weight_val,
                ) = train_test_split(
                    X,
                    y,
                    sample_weight,
                    test_size=self.validation_fraction,
                    stratify=stratify,
                    random_state=self._random_seed,
                )
```


### Describe your proposed solution

it would be sufficient to add an additional parameter to control whether or not to shuffle the data

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: aldder
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_25452: state: open
title: Issue building from source on Fedora 37 Python 3.11
body: Dear maintainers,
 I've recently updated my system to Fedora 37, shipping python 3.11.

I've created and activated a new python venv
```
python3 -m venv skldevenv/
source ~/skldevenv/bin/activate
```
then build main at commit  a7cd0ca44d0af64bc22a7213371e3c11d94dbbe8 in editable mode
```
pip install -e .
```
The build fails with the following error
```trpy
 python setup.py develop did not run successfully.
│ exit code: 1
╰─> [51 lines of output]
    Partial import of sklearn during the build process.
    running develop
    /tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    /tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
 
```

<details><summary>Full traceback</summary>

```trpy
Obtaining file:///home/cmarmo/software/scikit-learn
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy>=1.17.3 in /home/cmarmo/skldevenv/lib64/python3.11/site-packages (from scikit-learn==1.3.dev0) (1.24.1)
Requirement already satisfied: scipy>=1.3.2 in /home/cmarmo/skldevenv/lib64/python3.11/site-packages (from scikit-learn==1.3.dev0) (1.10.0)
Requirement already satisfied: joblib>=1.1.1 in /home/cmarmo/skldevenv/lib64/python3.11/site-packages (from scikit-learn==1.3.dev0) (1.2.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /home/cmarmo/skldevenv/lib64/python3.11/site-packages (from scikit-learn==1.3.dev0) (3.1.0)
Installing collected packages: scikit-learn
  Running setup.py develop for scikit-learn
    error: subprocess-exited-with-error
    
    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [51 lines of output]
        Partial import of sklearn during the build process.
        running develop
        /tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        /tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        Traceback (most recent call last):
          File "<string>", line 2, in <module>
          File "<pip-setuptools-caller>", line 34, in <module>
          File "/home/cmarmo/software/scikit-learn/setup.py", line 679, in <module>
            setup_package()
          File "/home/cmarmo/software/scikit-learn/setup.py", line 675, in setup_package
            setup(**metadata)
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 153, in setup
            return distutils.core.setup(**attrs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 148, in setup
            return run_commands(dist)
                   ^^^^^^^^^^^^^^^^^^
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
            dist.run_commands()
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
            self.run_command(cmd)
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 985, in run_command
            cmd_obj.ensure_finalized()
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
            self.finalize_options()
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/develop.py", line 52, in finalize_options
            easy_install.finalize_options(self)
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/easy_install.py", line 268, in finalize_options
            self._expand(
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/easy_install.py", line 1344, in _expand
            config_vars = self.get_finalized_command('install').config_vars
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 299, in get_finalized_command
            cmd_obj.ensure_finalized()
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
            self.finalize_options()
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/install.py", line 45, in finalize_options
            orig.install.finalize_options(self)
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/install.py", line 325, in finalize_options
            self.finalize_unix()
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/install.py", line 498, in finalize_unix
            self.select_scheme("posix_prefix")
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/install.py", line 528, in select_scheme
            return self._select_scheme(resolved)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/install.py", line 537, in _select_scheme
            setattr(self, attrname, scheme[key])
                                    ~~~~~~^^^^^
        KeyError: 'headers'
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [51 lines of output]
    Partial import of sklearn during the build process.
    running develop
    /tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    /tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "/home/cmarmo/software/scikit-learn/setup.py", line 679, in <module>
        setup_package()
      File "/home/cmarmo/software/scikit-learn/setup.py", line 675, in setup_package
        setup(**metadata)
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 148, in setup
        return run_commands(dist)
               ^^^^^^^^^^^^^^^^^^
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
        dist.run_commands()
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
        self.run_command(cmd)
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 985, in run_command
        cmd_obj.ensure_finalized()
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/develop.py", line 52, in finalize_options
        easy_install.finalize_options(self)
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/easy_install.py", line 268, in finalize_options
        self._expand(
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/easy_install.py", line 1344, in _expand
        config_vars = self.get_finalized_command('install').config_vars
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 299, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/command/install.py", line 45, in finalize_options
        orig.install.finalize_options(self)
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/install.py", line 325, in finalize_options
        self.finalize_unix()
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/install.py", line 498, in finalize_unix
        self.select_scheme("posix_prefix")
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/install.py", line 528, in select_scheme
        return self._select_scheme(resolved)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/tmp/pip-build-env-kiiuo3x3/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/install.py", line 537, in _select_scheme
        setattr(self, attrname, scheme[key])
                                ~~~~~~^^^^^
    KeyError: 'headers'
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
```

</details>

This is due to the fact that when creating the virtual environment setuptools 62.6.0 is installed
```
Python 3.11.1 (main, Jan  6 2023, 00:00:00) [GCC 12.2.1 20221121 (Red Hat 12.2.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>> setuptools.__version__
'62.6.0'
```
while scikit-learn asks for setuptools<60.0

Uninstalling setuptools
```
pip uninstall setuptools
```
and rebuilding
```
pip install -e .
```
fixes the issue.

Sharing here if anyone else experiences the same, and like me doesn't know setuptools is installed by default in the venv (maybe in Fedora?) ... and also naively assumes pyproject.toml directive is enough to rectify the installed version of setuptools.... 
user: cmarmo
issue_25448: state: open
title: MAINT Refactor Tree Cython class to support modularity
body: #### Reference Issues/PRs
Fixes: #25119 
Closes: #24746 
Closes: #24000

Supersedes: https://github.com/scikit-learn/scikit-learn/pull/25118

Requires https://github.com/scikit-learn/scikit-learn/pull/24678 to be merged first, since this is a fork of that branch.

This ends up being relatively large, and the below changes can be broken up probably into 2 PRs. One for splitting `Tree` -> `BaseTree` and `Tree`.

#### What does this implement/fix? Explain your changes.

1. Splits `Tree` class into a `BaseTree` and `Tree` class: The BaseTree does not assume any specifics on how nodes are split, how leaf nodes are set. This paves the path for enabling new trees such as: i) oblique trees, ii) causal trees and iii) quantile trees.
 - This would enable someone to easily subclass the Tree API without requiring specific data structures 
2. Adds new functions `_set_split_node()`, `_set_leaf_node()`, `_compute_feature()`, `_compute_feature_importances()` to allow subclasses of `BaseTree` to define any decision tree that generalizes in any one of those directions.
 - This would allow us to easily implement quantile-trees, honest-trees

#### Any other comments?
Cross-referencing: 
- https://github.com/scikit-learn/scikit-learn/pull/22754, which shows how this modularity can be useful
- https://github.com/scikit-learn/scikit-learn/issues/24990, which modularizes Splitter
- https://github.com/scikit-learn/scikit-learn/issues/24577, which modularizes Criterion

Reference for quantile-tree pseudocode demo: https://zillow.github.io/quantile-forest/auto_examples/plot_quantile_vs_standard_forest.html#sphx-glr-auto-examples-plot-quantile-vs-standard-forest-py

user: adam2392
issue_25440: state: open
title: [MRG] Fix to text_analytics/fetch_data.py not generating files
body: Noticed whilst going throught the tutorials that lxml had changed and was no longer generating the language files. Updated to use BeautifulSoup instead.

Validated files are generated correctly now.

Other changes to quotations in the file where generated by the commit hooks.

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: checkdgt
labels: Needs Reproducible Code: Issue requires reproducible code
issue_25434: state: open
title: Add `_asarray_fn` override to `check_array`
body: Closes https://github.com/scikit-learn/scikit-learn/issues/25433

user: fcharras
issue_25433: state: open
title: Open up `check_array` and `BaseEstimator._validate_data` to overriding `xp.asarray` with an additional callable parameter `asarray_fn`
body: ### Describe the workflow you want to enable

Some people (including @betatim @ogrisel @jjerphan and I) have been devising a plugin system that would open up `sklearn` estimators to other external implementations, and in particular implementations with GPU backends - see https://github.com/scikit-learn/scikit-learn/issues/22438 .

Some of the plugins we're considering can materialize the data in memory with an array library that is compatible with the Array API - namely [CuPy](https://github.com/scikit-learn/scikit-learn/issues/22438) and [dpctl.tensor](https://intelpython.github.io/dpctl/latest/docfiles/dpctl/dpctl.tensor_pyapi.html#dpctl-tensor-pyapi).

One thing we've found is that internally those plugins can benefit from using directly [`BaseEstimator._validate_data`](https://github.com/scikit-learn/scikit-learn/blob/98cf537f5/sklearn/base.py#L453) and [`check_array`](https://github.com/scikit-learn/scikit-learn/blob/d5fcb20b39c9b283d9485a9dcb4b2d554422e269/sklearn/utils/validation.py#L629) from `scikit-learn` to do the data acceptation and preparation step.

### Describe your proposed solution

To enable this it would be nice to be able to pass a `asarray_fn` to `check_array` and `_validate_data`, that would be called instead of `xp.asarray` in `_asarray_with_order` . This would enable the plugin to convert directly the input data to an array that the plugin supports (e.g. `cupy` or `dpctl.tensor`) while still benefiting from reusing existing validation code in `check_array`. 

The override can be necessary in case the `asarray` method from the array library implements a superset of the array api that is necessary for the plugin, but is currently not used by `check_array` because it's not part of the array api (for instance, the `order` argument isn't passed to `asarray` for array libraries other than `numpy`) 

user: fcharras
issue_25419: state: open
title: Add Warnings when trying to use estimator.fit()  functions with Pandas objects with different indices.
body: ### Describe the workflow you want to enable

Because sklearn uses numpy.array objects but accepts `pandas.DataFrame` / `pandas.Series` objects as input to its many functions, passing two pandas objects with different indices to a `estimator.fit()` function allows to unadvertedly work on shuffled X and y arrays. 

Let's construct an `X` `pandas.DataFrame` and an `y` `pandas.Series`.

```
import pandas as pd

x1 = [3, 3, 4, 5, 3, 5]
x2 = [0.1, 0.2, -2, -3, -3, -2]

y = ['a', 'a', 'b', 'b', 'b', 'b']

X_index = [1, 2, 3, 4, 5, 6]
y_index = [6, 5, 4, 3, 2, 1]

X = pd.DataFrame({'A': x1, 'B': x2}, index=X_index)
y = pd.Series(y, index=y_index)
```
Different pre-processing protocols (such as the use of `pandas.DataFrame.merge()`) can unadvertedly change the index of one of the two `pandas` objects (usually `X`, which is normally subject to more modifications than `y`).

Passing two incompatible indices like the ones above to an `estimator.fit()` function will yield no kind of warning. The model will be fit on two data objects which can potentially be in different orders, without expliciting the fact that the two indices are not equal. 

```
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
model.fit(X, y)
>>> LogisticRegression()

X.index == y.index
>>> array([False, False, False, False, False, False])
```

Checking if the two `pandas` objects' indices are equal and warning if they are not would allow users to understand that their `X` and `y` inputs may not be compatible. This can potentially save _a lot_ of time spent modeling on randomly shuffled data without warnings of any kind. Git knows I would've saved a lot of time. 

### Describe your proposed solution

For all fitting functions, implementing a simple check that  `X.index == y.index` is true for all cases before turning the `pandas` objects to `numpy`, and printing a warning informing that the indices are not compatible if it is not the case, urging the user to check that the order of their data is correct.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: MateoJacomeUPC
labels: Needs Decision: Requires decision
issue_25413: state: open
title: [Refactor Request Tree] Make sorting and splitting utility functions cimportable by including them in the pxd files
body: Hi,

I was wondering if it is possible to define the relevant splitter utility functions in the `.pxd` file so that it is cimportable from 3rd party applications?

## Motivation
3rd party applications will typically leverage the Cython code within scikit-learn even though it is not a publicly supported API. Refactoring the tree code has been discussed, but has been moving slowly due to lack of time from maintainers. In the meantime, one might want to use copy/modify some of the existing tree code to implement a new tree model. 

In the splitter, they will typically still use common functions such as "sort", "extract_nnz_binary_search", "extract_nnz_index_to_samples", "sparse_swap" (I may be missing others). 

## Solution
Just include their headers in the `sklearn.tree._splitter.pxd` file. This is similar to how common utility functions are also included in `sklearn.tree._utils.pxd`. 

This would be a reasonably small PR, with no side effects.
user: adam2392
labels: Needs Decision - API: Requires decision regarding API
issue_25412: state: open
title: InteractionTransformer
body: ### Describe the workflow you want to enable

The latest 1.2 release is full of great features, e.g., full column name support. This brings me to one of my most desired features regarding building strong and realistic linear models: Interactions!

It is currently very hard to add interaction terms between two feature groups, especially if they involve 1 to m transforms like OHE or SplineTransformers.

### Describe your proposed solution

An idea of @lorentzenchr that I try to summarize: Create the transforms like a ColumnTransformer,  but also adding interaction terms between columns generated by each transformer. The resulting columns could be glued to an other ColumnTransformer using `FeatureUnion`.

## Sketch of the API

```
InteractionTransformer(
    transformers,
    interaction_only=False
    include_bias=True,
    verbose_feature_names_out=True
)

# transformers: list of tuples
#     List of (name, transformer, columns) tuples specifying the transformer objects to be applied to subsets of the data.
#     Each transformer would specify a feature (group) that would interact with other transformers.
# interaction_only: If False, also return main effects
```

## Example 1: Interactions between two 1-m transforms

Here, we would let each dummy variable of "f1" interact with each spline basis of "f2":

```
InteractionTransformer(
    transformers=[
        ("f1_ohe", OneHotEncoder(drop="first"), ["f1"]),
        ("f2_spline", SplineTransformer(), ["f2"]),
    ]
)
```

## Example 2: Interaction between OHE and other features

Each column generated by OHE "f1" would interact with numeric feature "f2" and also with numeric feature "f3".

```
InteractionTransformer(
    transformers=[
        ("f1_ohe", OneHotEncoder(drop="first"), ["f1"]),
        ("f2_others", "passthrough", ["f2", "f3"]),
    ]
)
```

## Example 3: Interaction between two OHE, further features linear

```
interactor = InteractionTransformer(
    transformers=[
        ("f1_ohe", OneHotEncoder(drop="first"), ["f1"]),
        ("f2_ohe", OneHotEncoder(drop="first"), ["f2"]),
    ]
)

preprocessor = FeatureUnion(
  interactor,
  ColumnTransformer("other", "passthrough", ["f3", "f4"])
)

```


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: mayer79
issue_25395: state: open
title: Initializing new random instances for each estimator instead of passing around the same one
body: ### Describe the issue linked to the documentation

After reading the detailed and helpful section on ["Controlling randomness"](https://scikit-learn.org/stable/common_pitfalls.html#controlling-randomness), my understanding is that the benefit of passing an `np.random.RandomState` instance to each estimator is the different randomness for different folds during the CV process, which passing an integer would not provide. In the sample code, the random instance is defined globally and then passed to the `random_state` parameter of all function calls. The drawback of this is that the order of operations now matters (as mentioned in the documentation) and the addition of any code that uses the same globally defined random instance variable will advance the random number generator and change the results of any code that is run afterwards, which can be quite confusing and not ideal for reproducibility.

Instead of defining this random state globally and re-using it, would it not be more beneficial to pass a new instance to each estimator? It seems to me that this would combine the benefits of the CV exploring different randomness in each fold, and having the code reproducible regardless of insertion of new code that uses the same random instance.


### Suggest a potential alternative/fix

Instead of the current recommendation in the docs:

```py
rng = RandomState(0)  # from numpy.random import RandomState
X, y = make_classification(random_state=rng)
rf = RandomForestClassifier(random_state=rng)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=rng)
rf.fit(X_train, y_train).score(X_test, y_test)
```

I would expect this to combine the benefits I mentioned above:

```py
rng_init = 0
X, y = make_classification(random_state=RandomState(rng_init))
rf = RandomForestClassifier(random_state=RandomState(rng_init))
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=RandomState(rng_init))
rf.fit(X_train, y_train).score(X_test, y_test)
```

Apart from the increased verbosity, is there any downside to this approach (or is it even the same as passing an `int` so it has the same downsides)? If not, would you be open to a PR mentioning this in the docs? I am not that well-versed with pseudo random number generators in general, so while I know that the sequence generated by the number generator can be beneficial for some application, I don't know if there are any benefits to advancing this sequence as a global variable between different estimator call or if it is more important within each estimator and it is therefore fine to reinitialize the sequence as I suggest above.

And in this particular example, isn't only to the estimator we need to pass the `RandomState` instance (so the call to `make_classification` and `train_test_split` could receive an `int` instead which would make it less verbose)?


user: joelostblom
issue_25383: state: open
title: DOC document about the divergent factor of the Binomial loss
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
--> Fixes #25206 


#### What does this implement/fix? Explain your changes.
I did a documentation change where the deviance(=loss) given by the train_score_ is  double the actual log_loss which gave some non understandable results when using the train_score_ functions for the training data and a classic log_normal function for the test data. 
The fix i proposed is on the level of the documentary of the parameters of the loss of the GradientBoostingClassifier where i mentionned that loss(=deviance) might have a factor of two and that if we want the actual we need to take only half the loss.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Nebula1230
issue_25380: state: open
title: SVC and OneClassSVM fails to fit or have wrong fitted attributes with null sample weights
body: ### Describe the bug

*SVC().fit(X, y, w)* fails when the targets *y* are multiclass and the sample_weights *w* zero out one of the classes.
* Dense *X* produces incorrect arrays for *support_*, *n_support_*, and *dual_coef_* attributes, some with wrong dimension.
* Sparse *X* errors out when trying to construct sparse *dual_coef_* from incompatible arguments.

A warning is emitted (*e.g.*, "class label 0 specified in weight is not found"), but it does not indicate that the arrays on the trained SVC object are incorrect.

Seems to be a case that was not tested by PR #14286.

### Workaround

Replace the zero weights (or negative weights) with very small values like 1e-16. 

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.svm import SVC

X = np.array([[0., 0.], [1., 0.], [0., 1.]])   # or sp.csr_matrix(...)
y = [0, 1, 2]
w = [0., 1., 1.]                               # class 0 has zero weight
clf = SVC().fit(X, y, w)
```

### Expected Results

The fitted attributes should be
```
classes_: [0 1 2]
support_: [1 2]
support_vectors_: [[1. 0.]
                   [0. 1.]]
n_support_: [0 1 1]
dual_coef_: [[0. 0. 0.]
             [0. 1. -1.]]
```
assuming the 'arbitrary' values in *dual_coef_* are set to zero.

### Actual Results

For dense *X*, the fitted attributes are actually
```
classes_: [0 1 2]
support_: [0 1]              <-- should be [1 2]
support_vectors_: [[1. 0.]
                   [0. 1.]]
n_support_: [1 1]            <-- should be [0 1 1]
dual_coef_: [[ 1. -1.]]      <-- should be [[0. 0. 0.] [0. 1. -1.]]
```
For sparse *X* it raises
```
ValueError: indices and data should have the same size
```
with traceback
```
sklearn/svm/_base.py:252, in fit(self, X, y, sample_weight)
--> 252 fit(X, y, sample_weight, solver_type, kernel, random_seed=seed)

sklearn/svm/_base.py:413, in _sparse_fit(self, X, y, sample_weight, solver_type, kernel, random_seed)
--> 413     self.dual_coef_ = sp.csr_matrix(
    414         (dual_coef_data, dual_coef_indices, dual_coef_indptr), (n_class, n_SV)
    415     )

scipy/sparse/_compressed.py:106, in _cs_matrix.__init__(self, arg1, shape, dtype, copy)
--> 106 self.check_format(full_check=False)

scipy/sparse/_compressed.py:176, in _cs_matrix.check_format(self, full_check)
    174 # check index and data arrays
    175 if (len(self.indices) != len(self.data)):
--> 176     raise ValueError("indices and data should have the same size")
```


### Versions

```shell
System:
    python: 3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 08:55:37)  [Clang 14.0.6 ]
executable: miniconda3/bin/python
   machine: macOS-13.1-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.2.0
          pip: 22.3.1
   setuptools: 65.6.3
        numpy: 1.24.1
        scipy: 1.10.0
       Cython: 0.29.33
       pandas: 1.5.2
   matplotlib: 3.6.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: mkl
         prefix: libmkl_rt
       filepath: miniconda3/lib/libmkl_rt.dylib
        version: 2020.0.4
threading_layer: intel
    num_threads: 8

       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: miniconda3/lib/libomp.dylib
        version: None
    num_threads: 16
```

user: andrewdelong
labels: Hard: Hard level of difficulty
issue_25379: state: open
title: DOC Update IncrementalPCA example to actually use batches
body: #### What does this implement/fix? Explain your changes.

It wasn't clear to be how to actually use the API to train an IncrementalPCA. This updates the example, and gives the same results as before.

user: dsblank
issue_25360: state: open
title: DOC Example MLPRegressor as an autoencoder
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #25346
Fixes #25216

#### What does this implement/fix? Explain your changes.
This fix adds an example for MLPRegressor as an autoencoder

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: lfourniercm
issue_25356: state: open
title: [WIP] FEA Online Dictionary Learning with missing values
body: Continuation of https://github.com/scikit-learn/scikit-learn/pull/18492
user: jeremiedbb
issue_25330: state: open
title: ENH NearestNeighbors-like classes with metric="nan_euclidean" does not actually support NaN values
body: This PR fixes #25319.

As suggested by @glemaitre, I changed the `X, y` validation of `._fit` and then of `.kneighbors` and `.radius_neighbors` when `metric="nan_euclidean"` of `RadiusNeighborsMixin`, `KNeighborsMixin`, `NeighborsBase`. Consequently, changing the behavior of its heritage (`KNeighborsTransformer`, `RadiusNeighborsTransformer`,` KNeighborsClassifier`, `RadiusNeighborsClassifier`, `LocalOutlierFactor`, `KNeighborsRegressor`, `RadiusNeighborsRegressor`, `NearestNeighbors`).

I also updated the `NearestCentroid` class to follow this new validation. To make it work I had to change the validation of `sklearn.metrics.pairwise_distances_argmin` and `sklearn.metrics.pairwise_distances_argmin_min` as well (updating the docs now that it supports `metric=nan_euclidean'`).

As `KernelDensity` uses kd_tree or ball_tree to build index:
https://github.com/scikit-learn/scikit-learn/blob/98cf537f5c538fdbc9d27b851cf03ce7611b8a48/sklearn/neighbors/_kde.py#L48
It does not support `metrics='nan_euclidean'`, and I made no changes to it.
```python
from sklearn.neighbors import VALID_METRICS
for key in VALID_METRICS.keys():
    print(f"'nan_euclidean' in {key}:", 'nan_euclidean' in VALID_METRICS[key])
>>> 'nan_euclidean' in ball_tree: False
>>> 'nan_euclidean' in kd_tree: False
>>> 'nan_euclidean' in brute: True
```

Also, I added a test with the code used to report the issue by checking the behavior of the above classes.

user: vitaliset
issue_25326: state: open
title: feat: Support class weight for MLPClassifier
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes the issue https://github.com/scikit-learn/scikit-learn/issues/9113, and should also close the stalled PR https://github.com/scikit-learn/scikit-learn/pull/11723 
(The old PR hasn't been updated for more than 2 years, and it is difficult to rebase. That's why I have started this new PR.)


#### What does this implement/fix? Explain your changes.

- [x] Add sample_weight support

 

   - [x] Add support to multiply sample weight sample wise on loss functions:
     - [x] log_loss
     - [x] binary_log_loss
     - [x]  squared_loss
   - [x]  Add sample_weight parameter to fit and partial_fit
     - [x]  Propagate to relevant functions
     - [x]  Validate sample_weight for both MLPClassifier and MLPRegressor
     - [x]  Handle zero weights
   - [x]  Add tests
     - [x]  Test directly loss functions sample_weight invariance
     - [x]  Test fit validation of sample_weight

- [x]  Add class_weight support
  - [x]  Add class_weight parameter to fit and partial fit
  - [x]  Add tests for behavior correctness


#### Any other comments?
I am adding `class_weight` as a parameter to the fit and partial_fit functions because it makes more sense to do so than to make it an instance variable. I added an additional unit test to check the class weight the same way as `check_class_weight_classifiers` in `estimator_checks.py`

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: zshu115x
issue_25319: state: open
title: KNeighborsRegressor with metric="nan_euclidean" does not actually support NaN values
body: ### Describe the bug

[KNeighborsRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html) claims to support [these distance metrics](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.distance_metrics.html), which includes one called `"nan_euclidean"`, which presumably refers to [this metric](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.nan_euclidean_distances.html), which calculates "the euclidean distances in the presence of missing values." So, using this metric in `KNeighborsRegressor` should allow it to find nearest neighbors even in the presence of missing (NaN) values.

However, after setting `metric="nan_euclidean"`, the `fit()` method raises an error complaining that the "Input contains NaN". But this should not be an error, because I've chosen a distance metric that supports NaN values.

### Steps/Code to Reproduce

```py
X = [[0, 1], [1, np.nan], [2, 3,], [3, 5]]
y = [0, 0, 1, 1]
from sklearn.neighbors import KNeighborsRegressor
neigh = KNeighborsRegressor(n_neighbors=2, metric="nan_euclidean")
neigh.fit(X, y)
```

### Expected Results

No error is thrown.

### Actual Results

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-227-07ade1a53cbc>](https://localhost:8080/#) in <module>
      3 from sklearn.neighbors import KNeighborsRegressor
      4 neigh = KNeighborsRegressor(n_neighbors=2, metric="nan_euclidean")
----> 5 neigh.fit(X, y)
      6 neigh.predict([[1.5, 3]])

5 frames
[/usr/local/lib/python3.8/dist-packages/sklearn/neighbors/_regression.py](https://localhost:8080/#) in fit(self, X, y)
    211         self.weights = _check_weights(self.weights)
    212 
--> 213         return self._fit(X, y)
    214 
    215     def predict(self, X):

[/usr/local/lib/python3.8/dist-packages/sklearn/neighbors/_base.py](https://localhost:8080/#) in _fit(self, X, y)
    398         if self._get_tags()["requires_y"]:
    399             if not isinstance(X, (KDTree, BallTree, NeighborsBase)):
--> 400                 X, y = self._validate_data(X, y, accept_sparse="csr", multi_output=True)
    401 
    402             if is_classifier(self):

[/usr/local/lib/python3.8/dist-packages/sklearn/base.py](https://localhost:8080/#) in _validate_data(self, X, y, reset, validate_separately, **check_params)
    579                 y = check_array(y, **check_y_params)
    580             else:
--> 581                 X, y = check_X_y(X, y, **check_params)
    582             out = X, y
    583 

[/usr/local/lib/python3.8/dist-packages/sklearn/utils/validation.py](https://localhost:8080/#) in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
    962         raise ValueError("y cannot be None")
    963 
--> 964     X = check_array(
    965         X,
    966         accept_sparse=accept_sparse,

[/usr/local/lib/python3.8/dist-packages/sklearn/utils/validation.py](https://localhost:8080/#) in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
    798 
    799         if force_all_finite:
--> 800             _assert_all_finite(array, allow_nan=force_all_finite == "allow-nan")
    801 
    802     if ensure_min_samples > 0:

[/usr/local/lib/python3.8/dist-packages/sklearn/utils/validation.py](https://localhost:8080/#) in _assert_all_finite(X, allow_nan, msg_dtype)
    112         ):
    113             type_err = "infinity" if allow_nan else "NaN, infinity"
--> 114             raise ValueError(
    115                 msg_err.format(
    116                     type_err, msg_dtype if msg_dtype is not None else X.dtype

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
```

### Versions

```shell
System:
    python: 3.8.16 (default, Dec  7 2022, 01:12:13)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.10.147+-x86_64-with-glibc2.27

Python dependencies:
          pip: 22.0.4
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.21.6
        scipy: 1.7.3
       Cython: 0.29.32
       pandas: 1.3.5
   matplotlib: 3.2.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```
```

user: jameshfisher
issue_25258: state: open
title: Matthews correlation coefficient gives wrong results for single-class result
body: ### Describe the bug

for a single class input, (ie perfect correlation) MCC returns 0, when it should return 1.

since MCC is a special case of pearson correlation, it should fall back to pearson when appropriate

### Steps/Code to Reproduce

from sklearn.metrics import matthews_corrcoef
y_true = [1, 1, 1, 1]
y_pred = [1, 1, 1, 1] 
matthews_corrcoef(y_true, y_pred)
Out  [9]: 0.0

### Expected Results

output should be 1

### Actual Results

output is 0

### Versions

```shell
import sklearn; sklearn.show_versions()

System:
    python: 3.9.12 (main, Apr  5 2022, 01:53:17)  [Clang 12.0.0 ]
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 22.3.1
   setuptools: 65.5.1
      sklearn: 1.0.2
        numpy: 1.22.3
        scipy: 1.9.3
       Cython: 0.29.32
       pandas: 1.5.2
   matplotlib: 3.6.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
/Users/phild1/opt/anaconda3/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
```

user: dsphilli1
issue_25236: state: open
title: Allow for passing additional parameters to the estimator's `fit` method in `SequentialFeatureSelector`
body: ### Describe the workflow you want to enable

I would like to be able to pass sample weights to the `fit` method of the estimator in `SequentialFeatureSelector`. (`SelectFromModel` has this feature as well.)

Looking at the code, it seems to me that [`sklearn.model_selection._validation._fit_and_score`](https://github.com/scikit-learn/scikit-learn/blob/2e481f114169396660f0051eee1bcf6bcddfd556/sklearn/model_selection/_validation.py#L662), which is where the actual scoring takes place, already supports sample weights in cross-validation (i.e., takes care of passing only the weights for samples that are actually part of the fold).

### Describe your proposed solution

Based on my current understanding, `SequentialFeatureSelector.fit()` would need to get an additional argument `fit_params`  (cf. [`SelectFromModel.fit()`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html#sklearn.feature_selection.SelectFromModel.fit)) that is passed on to the `cross_val_score` function called in `SequentialFeatureSelector._get_best_new_feature_score`. Everything else appears to be in place already.

if that's all it takes, I'm happy to prepare a PR. But I might very well be overlooking issues or unintended consequences.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: ionicsolutions
issue_25217: state: open
title: FIX fix and improve OutputCodeClassifier
body: requires #25196

Rework the `OutputCodeClassifier` that suffers from several bugs:

- [x] use L1 distance instead of L2 distance during `predict`
- [x] implement loss-based decoding to align hard and probabilities in the future
- [x] avoid whenever possible collision in the `codebook`
- [x] do not rely on `decision_function` since the distance computed does not make sense in the hard decoding setup
- [x] add tests to ensure that the `predict` method is somehow working.

I define `decoding="hamming"` since it will work with any classifier (only `predict` is required).
user: glemaitre
issue_25216: state: open
title: Mention that MLPRegressor can function as an autoencoder
body: ### Describe the issue linked to the documentation

The MLPRegressor can function as an autoencoder by passing X as input and target (i.e. X == y).

```python
autoencoder = MLPRegressor(hidden_layer_sizes=(2,), activation="identity")
autoencoder.fit(X, X)
X_auto = autoencoder.predict(X)
```

I use PCA for dimensionality reduction a lot, but kept going to torch for autoencoders for comparison purposes. I thought adding an autoencoder to scikit-learn would be a good idea, but then it hit me that this is equivalent to an autoencoder. Yet this is not mentioned in the docs anywhere.

Relevant articles: 
Docs: https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html
Guide: https://scikit-learn.org/stable/modules/unsupervised_reduction.html#unsupervised-dimensionality-reduction

### Suggest a potential alternative/fix

I think mentioning this in the docs, or in the dimensionality reduction user guide could be a good idea. I'm happy to make a PR, but I'd like to get your input first.
user: stephantul
issue_25213: state: open
title: Segmentation fault occurs when KernelPCA is loaded after torchvision
body: ### Describe the bug

If KernelPCA is loaded after torchvision, then I get a segmentation fault. If I load them in the opposite order then I get no segmentaton fault.

### Steps/Code to Reproduce

```
import torchvision
from sklearn.decomposition import KernelPCA

import numpy as np

transformer = KernelPCA(n_components=7, kernel='rbf')
X_transformed = transformer.fit_transform(np.zeros([100,100]))
```

### Expected Results

No segmentation fault

### Actual Results

Segmentation fault (core dumped)

### Versions

```shell
System:
    python: 3.8.10 (default, Mar 15 2022, 12:22:08)  [GCC 9.4.0]
executable: /usr/bin/python3
   machine: Linux-5.4.0-77-generic-x86_64-with-glibc2.29

Python dependencies:
          pip: 22.1
   setuptools: 45.2.0
      sklearn: 0.24.2
        numpy: 1.22.4
        scipy: 1.4.1
       Cython: 0.29.23
       pandas: 1.4.3
   matplotlib: 3.5.2
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True


torchvision version: 0.10.0+cu102
```

user: LorenzoLuzi
issue_25210: state: open
title: ENH partial_dependece plot for HistGradientBoosting estimator fitted with `sample_weight`
body: ### Describe the workflow you want to enable

As partial dependence of a model at a point [is defined as an expectation](https://scikit-learn.org/stable/modules/partial_dependence.html#mathematical-definition), it should respect `sample_weight` if someone wishes to use it (for instance, when you know your `X` does not follow the distribution you are interested in).

#25209 tries to solve this for `method='brute'` when you have new `X`. For older tree-based models trained with sample_weights, [`method='recursion'`](https://nicolas-hug.com/blog/pdps) keeps track of the training `sample_weight` and calculates the `partial_dependece` with that into consideration.

But, as discussed during the implementation of `sample_weight` on the HistGradientBoosting estimators (https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-548295813), these models stores an attribute `_fitted_with_sw` and when `partial_dependece` with recursion is asked, it throws an error:

https://github.com/scikit-learn/scikit-learn/blob/205f3b76ef8500c90c346c6c6fb6f4e589368278/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py#L1142-L1148


### Describe your proposed solution

As discussed in https://github.com/scikit-learn/scikit-learn/issues/24872#issuecomment-1352354690, the big difference between other tree-based algorithms and HistGradientBoosting is that HistGradientBoosting does not save the `weighted_n_node_samples` when building the tree.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: vitaliset
issue_25206: state: open
title: Unclear train_score_ attribute description for GradientBoostingClassifier
body: ### Describe the issue linked to the documentation

I recently trained a binary gradient boosting model using the GradientBoostingClassifier. I wanted to plot the intermediate losses at each iteration on the test set while evaluating my results and stumbled upon the `train_score_` attribute which is described as `The i-th score train_score_[i] is the deviance (= loss) of the model at iteration i on the in-bag sample.`. 
However, when plotting my test losses (calculated using normal log_loss using probabilities based on the `staged_predict_proba(X)` function) the training scores were not in the expected [0,1) interval and the resulting plot did not make a lot of sense (I would love a test set with the loss curve :D). Following the plot:

![loss_without_scalled_test_loss](https://user-images.githubusercontent.com/44102842/208318601-d3c3d372-f91e-4694-9a14-e51889a90e85.png)


After some digging, I found out that, at least for binomial log_loss, deviance is scaled by a factor of 2 using the following formula:

```
-2 * np.mean((y * raw_predictions) - np.logaddexp(0, raw_predictions))
```


Knowing this, the fix for the problem is very easy but at least for me took quite some time to find the scaling factor. For completion, the resulting plot makes much more sense if we scale the test loss by the same factor of 2:

![loss_with_scaled_test_loss](https://user-images.githubusercontent.com/44102842/208318607-d48c0d63-635a-49b6-934f-19a3069c9f5d.png)


### Suggest a potential alternative/fix

I would suggest adding a hint to the `train_scores_` attribute that the deviance is not necessarily the loss but could be a scaled version of the loss and dependent on the loss function employed. In addition, I would somehow change the `deviance (= loss)` part of the current documentation as the equality sign leaves room for interpretation. 

I did not open a pull request with a proposed change since my problem is quite specific to the binomial log loss/deviance and might not be important enough for a dedicated hint. 
user: Grimaldus12
issue_25198: state: open
title: Sparse data representations results in worse models than dense data for some classifiers
body: ### Describe the bug

Using scipy sparse matrices with sklearn LogisticRegression greatly improves speed and therefore is desirable in many scenarios.

However, it appears that sparse versus dense data representations yield different (worse) results for some sklearn classifiers.

My perhaps naive assumption is that sparse versus dense is just a method of representing the data and operations performed on the sparse or dense data (including model training) should yield identical or nearly identical results.

A notebook gist looking at sparse versus dense results for nine solvers can be found here: https://gist.github.com/mmulthaup/db619d8b5ea4baf4a00153b055a7e9a8

### Steps/Code to Reproduce

```py
#Minimal example
import sklearn
import scipy
import numpy as np
 
#Artificial data
y = np.repeat(1,100).tolist()+np.repeat(0,100).tolist()
X = np.concatenate([scipy.stats.poisson.rvs(0.2,size=[100,1000]),scipy.stats.poisson.rvs(0.1,size=[100,1000])])
 
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(
    X, y, test_size=0.5, random_state=42)
X_train_sparse = scipy.sparse.bsr_array(X_train)
X_test_sparse = scipy.sparse.bsr_array(X_test)
 
#Modeling
model = sklearn.linear_model.LogisticRegression(solver="saga",random_state=42,max_iter=4000)
dense_scores = model.fit(X_train,y_train).predict_proba(X_test)[:,1]
sparse_scores = model.fit(X_train_sparse,y_train).predict_proba(X_test_sparse)[:,1]

print(f"Dense AUC: {round(sklearn.metrics.roc_auc_score(y_test,dense_scores),3)}") #Dense AUC: 1.0
print(f"Sparse AUC: {round(sklearn.metrics.roc_auc_score(y_test,sparse_scores),3)}") #Sparse AUC: 0.584
```

### Expected Results

Dense AUC: 1.0
Sparse AUC: 1.0

### Actual Results

Dense AUC: 1.0
Sparse AUC: 0.584


### Versions

```shell
Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7fc557a3f310>
Traceback (most recent call last):
  File "/databricks/python/lib/python3.9/site-packages/threadpoolctl.py", line 400, in match_module_callback
    self._make_module_from_path(filepath)
  File "/databricks/python/lib/python3.9/site-packages/threadpoolctl.py", line 515, in _make_module_from_path
    module = module_class(filepath, prefix, user_api, internal_api)
  File "/databricks/python/lib/python3.9/site-packages/threadpoolctl.py", line 606, in __init__
    self.version = self.get_version()
  File "/databricks/python/lib/python3.9/site-packages/threadpoolctl.py", line 646, in get_version
    config = get_config().split()
AttributeError: 'NoneType' object has no attribute 'split'

System:
    python: 3.9.5 (default, Nov 23 2021, 15:27:38)  [GCC 9.3.0]
executable: /local_disk0/.ephemeral_nfs/envs/pythonEnv-0c18486f-171e-4bd6-9fb6-691f3a2da533/bin/python
   machine: Linux-5.4.0-1088-aws-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.2.0
          pip: 21.2.4
   setuptools: 61.2.0
        numpy: 1.23.5
        scipy: 1.9.3
       Cython: 0.29.28
       pandas: 1.5.2
   matplotlib: 3.5.1
       joblib: 1.2.0
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       filepath: /local_disk0/.ephemeral_nfs/envs/pythonEnv-0c18486f-171e-4bd6-9fb6-691f3a2da533/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
         prefix: libgomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 16

       filepath: /local_disk0/.ephemeral_nfs/envs/pythonEnv-0c18486f-171e-4bd6-9fb6-691f3a2da533/lib/python3.9/site-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
         prefix: libopenblas
       user_api: blas
   internal_api: openblas
        version: 0.3.18
    num_threads: 16
threading_layer: pthreads
   architecture: Haswell
```

user: mmulthaup
issue_25193: state: open
title: StratifiedKFold and StratifiedGroupKFold for multilabel classification
body: ### Describe the workflow you want to enable

Currently, these two functionalities only support binary/multiclass classification. I am looking for similar functionalities (maybe not even k fold, a train_test_split also suffices) in multilabel classification. 

### Describe your proposed solution

For the multilabel version of stratification, [this](https://github.com/trent-b/iterative-stratification) and [this](http://scikit.ml/api/skmultilearn.model_selection.iterative_stratification.html#module-skmultilearn.model_selection.iterative_stratification) libraries seem already provided a solution. Adding grouping to them should be similar to how `StratifiedGroupKFold` adds to `StratifiedKFold`.

### Describe alternatives you've considered, if relevant

See above for existing libraries developed based upon Sklearn.

### Additional context

_No response_
user: jasperhyp
issue_25192: state: open
title: FIX improve error message when no samples are available in mutual information
body: #### Reference Issues/PRs
closes #25179

check if instances are left after masking of unique labels
user: makoeppel
issue_25187: state: open
title: Early Stopping for GridSearchCV, RandomizedSearchCV
body: ### Describe the workflow you want to enable

- I have a custom model implementing the BaseEstimator, for which I am using scikit-learn's hyperparameter searches.
- I am running an exhaustive grid search, all possible parameters for my model.
- If one parameter setting gives a sufficiently high accuracy (e.g. F1 = 1.0), I wish to stop the grid search and free the compute for other runs.

### Describe your proposed solution

GridSearchCV, RandomizedSearchCV, and others should have an early stopping criteria. I should be able to specify a threshold accuracy, such that when the value returned by the scoring function passes this threshold, other jobs are stopped.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

* As parameter search happens in parallel, it may require some rewrite of the parallelization code so that results can be checked as they return: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/model_selection/_search.py#L822 
See, for instance, https://github.com/joblib/joblib/issues/356#issuecomment-224508490 
* I am opening this feature request to start a discussion; I am happy to do the development and submit a pull request if we agree on what should be built.
user: bwindsor22
issue_25179: state: open
title: Improve error handling in _mutual_info.py
body: ### Describe the workflow you want to enable

Hi all,

I am currently trying to run the following code:

```python
from sklearn.feature_selection import mutual_info_classif

a = [[1,0,1],[0,1,1]]
b = [0,1]

mutual_info_classif(a,b)
```
Which fails with:

```
Traceback (most recent call last):
  File "test.py", line 6, in <module>
    mutual_info_classif(a,b)
  File "python3.10/site-packages/sklearn/feature_selection/_mutual_info.py", line 465, in mutual_info_classif
    return _estimate_mi(X, y, discrete_features, True, n_neighbors, copy, random_state)
  File "python3.10/site-packages/sklearn/feature_selection/_mutual_info.py", line 301, in _estimate_mi
    mi = [
  File "python3.10/site-packages/sklearn/feature_selection/_mutual_info.py", line 302, in <listcomp>
    _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)
  File "python3.10/site-packages/sklearn/feature_selection/_mutual_info.py", line 165, in _compute_mi
    return _compute_mi_cd(x, y, n_neighbors)
  File "python3.10/site-packages/sklearn/feature_selection/_mutual_info.py", line 140, in _compute_mi_cd
    kd = KDTree(c)
  File "sklearn/neighbors/_binary_tree.pxi", line 955, in sklearn.neighbors._kd_tree.BinaryTree.__init__
  File "python3.10/site-packages/sklearn/utils/validation.py", line 805, in check_array
    raise ValueError(
ValueError: Found array with 0 sample(s) (shape=(0, 1)) while a minimum of 1 is required.
```

The code does fail because in the file `_mutual_info.py` all instances with a unique label are removed before the KDTree is called:

```python
n_samples = c.shape[0]
c = c.reshape((-1, 1))

radius = np.empty(n_samples)
label_counts = np.empty(n_samples)
k_all = np.empty(n_samples)
nn = NearestNeighbors()
for label in np.unique(d):
    mask = d == label
    count = np.sum(mask)
    if count > 1:
        k = min(n_neighbors, count - 1)
        nn.set_params(n_neighbors=k)
        nn.fit(c[mask])
        r = nn.kneighbors()[0]
        radius[mask] = np.nextafter(r[:, -1], 0)
        k_all[mask] = k
    label_counts[mask] = count

# Ignore points with unique labels.
mask = label_counts > 1
n_samples = np.sum(mask)
label_counts = label_counts[mask]
k_all = k_all[mask]
c = c[mask]
radius = radius[mask]
kd = KDTree(c)
```

I need to understand what the code does before I could debug my test script. A better error would helped here to find the problem faster.

### Describe your proposed solution

I guess the error could be improved here by checking if `c` is empty before calling KDTree:

```python
from ..utils.validation import check_array, check_X_y, _num_samples

...
radius = radius[mask]
if _num_samples(c) == 0:
    raise ValueError(
            "Found array with 0 sample(s) after masking"
            " points with unique labels. Ensure to have at least" 
            " two instances with the same label." 
    )
kd = KDTree(c)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: makoeppel
issue_25148: state: open
title: [MRG] Implement predict_proba() for OutputCodeClassifier
body: #### Reference Issues/PRs
None found, albeit it was briefly mentioned in #389

#### What does this implement/fix? Explain your changes.
In its present state, `sklearn.multiclass.OutputCodeClassifier()` is only able to `predict()`. Having access to the underlying class scores/probabilities is required for a good part of validation metrics. Those can also be used for model calibration and improved decision making (though that might be considered redundant for the ECOC classifier usecases).

#### What does it do / how does it work?
This PR implements the (faux) probability estimates using the same base estimators' output <-> codebook euclidean distances that are currently used by the `predict()` method using inverse distance weighting. Distance calculation is moved to the new `predict_proba()` method, and `predict()` is adjusted accordingly.

#### Any other comments?
**Possible cons:**

The probabilistic interpretation of said distances might be questionable. However, lots of other classification algorithms that do implement `predict_proba()` aren't guaranteed to have its output well-calibrated either.

**QA:**

- Precautions taken to avoid zero division in IDW calculation.
- ~~A non-regression test to ensure the `predict()` results are the same as before is provided,~~ as well as the sanity test of class scores summing to 1.
- pytest/black/flake8/mypy tests passed.
- The scores returned by the new `predict_proba()` appear to yield sane OvR ROC and PR curves, as well as calibration curves of an acceptable shape.
user: dx2-66
issue_25143: state: open
title: PCA - Add Loadings Property
body: If these are the loadings, can we surface them through a high level property? https://stackoverflow.com/a/42046659/670433






user: s2t2
issue_25119: state: open
title: MAINT Split Tree into a `BaseTree` and a `Tree` subclass to allow easier inheritance 
body: ### Summary
#24678 introduces a modularization of `Criterion` to allow different criterion to be used with the same classes.
#25101 introduces a modularization of `Splitter` to allow different types of of splits to be computed.

Now comes the time to also modularize the `Tree` class. A good `Tree` class should enable oblique splits, causal leaf nodes (i.e. leaf nodes set differently from split nodes), quantile trees (leaf nodes set differently from split nodes) and unsupervised trees. Note another feature of causal trees is 'honesty', which should be easier to add after this issue is resolved.

### Proposed improvement

We will have the following improvements:

1. Refactor `tree._add_node()` to set the split node and leaf node differently.
2. Refactor to have a 'splitptr' for `SplitRecord`, which allows for generalizations of the SplitRecord.
3. Separate `Tree` into generic and abstract base functions for `BaseTree` and specific supervised axis-aligned functions for `Tree`

Once the changes are made, one should verify:

1. If `tree` submodule's Cython code still builds (i.e. `make clean` and then `pip install --verbose --no-build-isolation --editable .` should not error out)
2. verify unit tests inside `sklearn/tree` all pass
4. verify that the asv benchmarks do not show a performance regression.

`asv continuous --verbose --split --bench RandomForest upstream/main <new_branch_name>` and then for side-by-side comparison `asv compare main <new_branch_name>`

### Reference

As discussed in #24577 , I wrote up a doc on proposed improvements to the tree submodule that would:
1. make it easier for 3rd party packages to subclass existing sklearn tree code and
2. make it easier for sklearn itself to make improvements to the tree code with many of the modern improvements to trees

cc: @jjerphan 


user: adam2392
labels: Refactor: Code refactor
issue_25118: state: open
title: MAINT Introduce `BaseTree` as a base abstraction for `Tree`
body: #### Reference Issues/PRs
Fixes: #25119 
Closes: #24746 
Closes: #24000

Requires https://github.com/scikit-learn/scikit-learn/pull/24678 to be merged first, since this is a fork of that branch.

This ends up being relatively large, and the below changes can be broken up probably into 2 PRs. One for splitting `Tree` -> `BaseTree` and `Tree`.

#### What does this implement/fix? Explain your changes.

1. Splits `Tree` class into a `BaseTree` and `Tree` class: The BaseTree does not assume any specifics on how nodes are split, how leaf nodes are set. This paves the path for enabling new trees such as: i) oblique trees, ii) causal trees and iii) quantile trees.
2. Adds new functions `_set_split_node()`, `_set_leaf_node()`, `_compute_feature()`, `_compute_feature_importances()` to allow subclasses of `BaseTree` to define any decision tree that generalizes in any one of those directions.

#### Any other comments?
Cross-referencing: 
- https://github.com/scikit-learn/scikit-learn/pull/22754, which shows how this modularity can be useful
- https://github.com/scikit-learn/scikit-learn/issues/24990, which modularizes Splitter
- https://github.com/scikit-learn/scikit-learn/issues/24577, which modularizes Criterion

user: adam2392
issue_25107: state: open
title: Make it easier to overload the `fit_methods` in `check_param_validation`
body: We introduced `check_param_validation`. This check could be useful to other third-party libraries. However, it will be limited to the fit methods defined in the scikit-learn API.

In imbalanced-learn, we define `fit_resample`. Such a method would not be called the `check`. I am wondering if we could expose the list of the fit methods and make it easy to register a method to be tested.
user: glemaitre
labels: Needs Decision - Include Feature: Requires decision regarding including feature
Validation: related to input validation
issue_25101: state: open
title: MAINT Refactor `Splitter` into a `BaseSplitter`
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #24990

#### What does this implement/fix? Explain your changes.
Adds the BaseSplitter class as an abstract class to be inherited by Splitter class which is now modularized without assuming "supervised learning" setting. To achieve this, moderate refactoring was done on Criterion by separating resetting of the index pointers from initialization process by creating `set_sample_pointers` method that is called by child classes of the Criterion.

This change is backwards compatible.

ASV Benchmark result
```sh
(ndd-split) jshinm@jshinm-OMEN-by-HP-Laptop-16-b0xxx:~/Desktop/workstation/sklearn/asv_benchmarks$ asv compare main split

All benchmarks:

       before           after         ratio
     [743fe8e5]       [b83cbbc7]
     <main>           <split>   

             190M             190M     1.00  ensemble.RandomForestClassifierBenchmark.peakmem_fit('dense', 1)
             423M             423M     1.00  ensemble.RandomForestClassifierBenchmark.peakmem_fit('sparse', 1)
             190M             190M     1.00  ensemble.RandomForestClassifierBenchmark.peakmem_predict('dense', 1)
             407M             407M     1.00  ensemble.RandomForestClassifierBenchmark.peakmem_predict('sparse', 1)
       4.93±0.02s       5.08±0.01s     1.03  ensemble.RandomForestClassifierBenchmark.time_fit('dense', 1)
       6.30±0.01s          6.32±0s     1.00  ensemble.RandomForestClassifierBenchmark.time_fit('sparse', 1)
        131±0.9ms        131±0.7ms     0.99  ensemble.RandomForestClassifierBenchmark.time_predict('dense', 1)
          847±2ms          855±4ms     1.01  ensemble.RandomForestClassifierBenchmark.time_predict('sparse', 1)
  0.7552784412549299  0.7552784412549299     1.00  ensemble.RandomForestClassifierBenchmark.track_test_score('dense', 1)
  0.8656423941766682  0.8656423941766682     1.00  ensemble.RandomForestClassifierBenchmark.track_test_score('sparse', 1)
  0.9961421915584339  0.9961421915584339     1.00  ensemble.RandomForestClassifierBenchmark.track_train_score('dense', 1)
  0.9996123288718864  0.9996123288718864     1.00  ensemble.RandomForestClassifierBenchmark.track_train_score('sparse', 1)
```

Test Machine Spec
```sh
os [Linux 5.15.0-56-generic]
arch [x86_64]
cpu [11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz]
num_cpu [16]
ram [65483276]
```

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: jshinm
issue_25079: state: open
title: RFC refactoring of PartialDependenceDisplay
body: I would like to discuss the possibility to refactor `PartialDependenceDisplay` to reduce its usability. The idea behind this refactoring is:

- make it consistent with other displays: calling `PartialDependenceDisplay` is equivalent to a call to `partial_dependence`.
- it will simplify greatly the code base: the codebase was already refactored but this is still difficult to understand logic where axes are interacting with each other.
- keep the figure close to standard `matplotlib`: it allows a user that knows `matplotlib` to quickly modify and tune the figure without knowing some implementation details enforced by `scikit-learn`.

### Sharing y-axis

One main reason for having `PartialDependenceDisplay` doing several plots is mainly for setting a common y-axis. However, `matplotlib` allows such a feature when creating a figure:

```python
features = ["age", "sex"]
fig, axs = plt.subplots(figsize=(12, 4), ncols=len(features), sharey=True)
for feat, ax in zip(features, axs):
    pd_values = partial_dependence(
        model,
        X,
        features=feat,
        categorical_features=categorical_features,
    )
    if feat in categorical_features:
        ax.bar(
                pd_values["values"][0], pd_values["average"][0]
            )
    else:
        ax.plot(
            pd_values["values"][0], pd_values["average"][0]
        )
```

![image](https://user-images.githubusercontent.com/7454015/204843253-6983838c-2ce7-494c-98d5-e0a17c2a5048.png)

Here, the call could be done with `PartialDependenceDisplay` instead of the call to `partial_dependence` + plotting.

In this case, a user can still manipulate the difference axes using `axs` and make any usual matplotlib.

### Surprising API

With the current `PartialDependenceDisplay`, we will have some parameters specific to our settings: `ncols` to set the number of columns and fitted attribute `bounding_axis_` and `axes_`. The latest attributes are required if someone is giving an axis, for instance, to provide a certain figure size:

```python
fig, ax = plt.subplots(figsize=(12, 4))
disp = PartialDependenceDisplay.from_estimator(
    ...
    ax=ax,
)
```

I personally find it more intuitive to have `ax` containing all the axes without picking up in the scikit-learn documentation, which way the plot is organized.

### Cons

The only thing that could be more difficult is to plot 1D and 2D PD at the same time. Since one wants to share y-axis for 1D plot, this is indeed not a desired feature to have for the 2D PD. Therefore, it requires our user to know how to share y-axis only for some of the axis.

At the same time, it could be argued that having a mixed type of 1D and 2D plots on the same figure should not be a feature :) Those should be on separate figures.

### Proposed API

The new API would force the user to materialize a `for` loop to handle each individual plot as:

```python
features = ["age", "sex"]
fig, axs = plt.subplots(figsize=(12, 4), ncols=len(features), sharey=True)
for feat, ax in zip(features, axs):
    PartialDependenceDisplay.from_estimator(
        model,
        X,
        feature=feat,
        categorical_features=categorical_features,
        ax=ax,
    )
user: glemaitre
issue_25068: state: open
title: RFC Consolidating synchronous communication to one venue
body: Tl;DR: scikit-learn development discussions now take place in [the #development channel on discord](https://discord.com/channels/731163543038197871/1046822941586898974). Please don't use the [gitter channel](https://gitter.im/scikit-learn/dev) anymore.

Discord invite link: https://discord.gg/ZDZzqwESay

---

The goal of this issue is to "shop around" an idea from the monthly meeting (28 Nov 2022). To gather feedback and impressions on the proposal.

There are several real-time/synchronous communication platforms in use for the scikit-learn project. There are several gitter channels and a discord instance with several channels. Combined with a (perceived) lack of a clear answer to "Where should I go for my synchronous communication?" this leads to lots of channels that appear "empty" at first glance (think "curse of high dimensions"). To make matters worse, if a venue appears empty people will try new/other venues to reach the people they are trying to reach. Which further increases the possible venues, further reducing the probability of observing activity in any one of the venues.

This topic was discussed at the monthly meeting (28 November 2022) and the feeling was that the status quo is worth improving. There was consensus that reducing (to one) the number of venues would be useful. There was no clear preference for which of the many platforms to select. There are pros and cons to each of them, especially considering the various use cases (sync communication to coordinate repository activity, voice/video channels for sprints, administrative chat).

However, there is a concrete proposal that didn't result in big opposition in the call. It tries to serve all of the use-cases, but of course tradeoffs had to be made. The proposal is to consolidate to "discord with a revamped channel list".

Concretely this means to point people towards the existing discord instance, away from other chat platforms, and to restructure the channels to make it easier to know which one to use when. A proposed list of channels:
* a "read me first" channel: this channel gives people an introduction to where they are, overview of channels, rules, expectations
	* for example making clear that the "users" channel isn't a "hotline to the developers". There are better places for support style discussions (we should link them).
* a "welcome/introduce yourself" channel: useful for people to say "Hi 👋!". Especially during online sprints
* a "development" channel: a place for people to synchronously discuss things that are going on in the repository right now. People who are (very) active on the repository could use it to discuss where they need help, how to resolve a problem, get help with a review, announce what they are focusing on or that they will have reduced availability for a while. Basically the kind of topics you'd talk about when in an office with others you work with.
* a "users" channel: this is the place for people using scikit-learn to talk to each other. 
* a "admin" channel: a (private) place for administrative tasks related to the discord instance
* a selection of voice channels (with explicit accompanying text rooms): these are for ad-hoc voice chats, to create "tables" during sprints and other occasions where you need several ad-hoc video+voice communication channels.


The goal of this issue is to "shop around" the idea from the monthly meeting. This means, gathering feedback and impressions on the proposal. Depending on what happens the next steps will hopefully become clearer. Keep in mind that there was support for making a change/the current situation is not great. This means that we should try and use this issue to iterate towards something that we can do, even if it is not perfect or might need tweaking in the future. A change that is too ambitious or needs too many resources to implement is a change that might never happen. Similarly, not making a decision is a form of passive decision making for the status quo. It is probably better to actively decide either way than let it linger.


---

I wasn't sure if I should open this issue here or on github.com/scikit-learn/administrative. If that would have been a better place, please someone move this issue there.
user: betatim
issue_25050: state: open
title: Davies-Bouldin Index implementation is incomplete
body: ### Describe the workflow you want to enable

The DB-Index has two parameters, p and q, where p is the power of the Minkowski norm used to compute the distance of cluster centers, while q controls the power mean for the "radius" / scatter. p=q=2 is probably the most meaningful combination. The current implementation appears to be hard-coded to p=2, q=1

### Describe your proposed solution

Add `p` and `q` parameters, default to what the current implementation uses.

Allow passing known cluster centers with a `center=` parameter, instead of recomputing them inside the function.
Enhance the documentation that this index is most suited for k-means clustering, because it uses cluster centroids.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: kno10
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_25038: state: open
title: Instability in fastica for float32
body: Originally seen in https://github.com/scikit-learn/scikit-learn/issues/24131#issuecomment-1208091119 for more details on a single random seed and only for Atlas but I have managed to reproduce with OpenBLAS see below.

For now we have a work-around for the CI in #24198 but it is still an issue and in an ideal world, someone would investigate and try to see whether it is fixable.

Other possibly related issues and PRs are mentioned in https://github.com/scikit-learn/scikit-learn/pull/24198#issuecomment-1252616127: https://github.com/scikit-learn/scikit-learn/issues/2735 and https://github.com/scikit-learn/scikit-learn/pull/2738.

Here is a snippet that reproduces the same problem on OpenBLAS:

```py
import numpy as np
from scipy import stats
from sklearn.decomposition import fastica
import sys

global_random_seed = 20
global_dtype = np.float32


def center_and_norm(x, axis=-1):
    x = np.rollaxis(x, axis)
    x -= x.mean(axis=0)
    x /= x.std(axis=0)


rng = np.random.RandomState(global_random_seed)
n_samples = 1000
# Generate two sources:
s1 = (2 * np.sin(np.linspace(0, 100, n_samples)) > 0) - 1
s2 = stats.t.rvs(1, size=n_samples, random_state=global_random_seed)
s = np.c_[s1, s2].T
center_and_norm(s)
s = s.astype(global_dtype)
s1, s2 = s

# Mixing angle
phi = 0.6
mixing = np.array([[np.cos(phi), np.sin(phi)], [np.sin(phi), -np.cos(phi)]])
mixing = mixing.astype(global_dtype)
m = np.dot(mixing, s)
center_and_norm(m)

algo = 'deflation'
nl = 'logcosh'
whiten = 'arbitrary-variance'

problematic_random_state = 13441

k_, mixing_, s_ = fastica(
    m.T, fun=nl, whiten=whiten, algorithm=algo, random_state=problematic_random_state
)
```

You get a warning with a division by zero:
```
/home/lesteve/dev/scikit-learn/sklearn/decomposition/_fastica.py:89: RuntimeWarning: invalid value encountered in divide
  w1 /= np.sqrt((w1**2).sum())
```

And then the following traceback because NaNs are passed into a BLAS routine eventually:

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In [1], line 39
     35 whiten = 'arbitrary-variance'
     37 problematic_random_state = 13441 # both for openblas and atlas (also numpy < and >= 1.23)
---> 39 k_, mixing_, s_ = fastica(
     40     m.T, fun=nl, whiten=whiten, algorithm=algo, random_state=problematic_random_state
     41 )

File ~/dev/scikit-learn/sklearn/decomposition/_fastica.py:322, in fastica(X, n_components, algorithm, whiten, fun, fun_args, max_iter, tol, w_init, whiten_solver, random_state, return_X_mean, compute_sources, return_n_iter)
    174 """Perform Fast Independent Component Analysis.
    175 
    176 The implementation is based on [1]_.
   (...)
    308        pp. 411-430.
    309 """
    310 est = FastICA(
    311     n_components=n_components,
    312     algorithm=algorithm,
   (...)
    320     random_state=random_state,
    321 )
--> 322 S = est._fit_transform(X, compute_sources=compute_sources)
    324 if est._whiten in ["unit-variance", "arbitrary-variance"]:
    325     K = est.whitening_

File ~/dev/scikit-learn/sklearn/decomposition/_fastica.py:683, in FastICA._fit_transform(self, X, compute_sources)
    680 else:
    681     self.components_ = W
--> 683 self.mixing_ = linalg.pinv(self.components_, check_finite=False)
    684 self._unmixing = W
    686 return S

File ~/miniconda3/lib/python3.9/site-packages/scipy/linalg/_basic.py:1304, in pinv(a, atol, rtol, return_rank, check_finite, cond, rcond)
   1231 """
   1232 Compute the (Moore-Penrose) pseudo-inverse of a matrix.
   1233 
   (...)
   1301 
   1302 """
   1303 a = _asarray_validated(a, check_finite=check_finite)
-> 1304 u, s, vh = _decomp_svd.svd(a, full_matrices=False, check_finite=False)
   1305 t = u.dtype.char.lower()
   1306 maxS = np.max(s)

File ~/miniconda3/lib/python3.9/site-packages/scipy/linalg/_decomp_svd.py:133, in svd(a, full_matrices, compute_uv, overwrite_a, check_finite, lapack_driver)
    131     raise LinAlgError("SVD did not converge")
    132 if info < 0:
--> 133     raise ValueError('illegal value in %dth argument of internal gesdd'
    134                      % -info)
    135 if compute_uv:
    136     return u, s, v

ValueError: illegal value in 4th argument of internal gesdd
```
user: lesteve
issue_25000: state: open
title: Feature request: an additional config context for forcing conversion toward a specific Array API-compatible array library.
body: ### Describe the workflow you want to enable

Per https://github.com/scikit-learn/scikit-learn/pull/22554 such workflow is possible:

```python
from sklearn.datasets import make_classification
from sklearn import config_context
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
import cupy.array_api as xp

X_np, y_np = make_classification(random_state=0)
X_cu = xp.asarray(X_np)
y_cu = xp.asarray(y_np)
X_cu.device


with config_context(array_api_dispatch=True):
    lda = LinearDiscriminantAnalysis()
    X_trans = lda.fit_transform(X_cu, y_cu)
```

(see https://scikit-learn.org/dev/modules/array_api.html )

I would like to additonally enable following workflow:

```python
X_np, y_np = make_classification(random_state=0)

with config_context(array_api_dispatch=True, array_api_namespace=xp):
    lda = LinearDiscriminantAnalysis()
    X_trans = lda.fit_transform(X_np, y_np)
```

where `X_trans` are the same in both examples, and in both examples compute has been done with `cupy`.  The difference is that we pass numpy inputs and rely on the estimator to convert using the appropriate array namespace.

The added value here for the user is that the `check_array` function could be used to load arbitrary inputs to the array library relying on its `xp.asarray` method, in a way that is consistent to `sklearn.utils.validation.check_array` requirements accross all array libraries.

Currently, a user that would have data formated as, let's say, a list of lists, has two choices:

- either pass the list of list directly to `lda.fit`, but then since the list of list does not have a `__array_namespace__` attribute it will fallback to numpy array namespace
- either convert first the input to `cupy`, but then, unless the user re-implements methodically `check_array` with `cupy` conversion, the behavior of the conversion might differ from what `check_array` would do when converting to `numpy`, and anyway reimplementing `check_array` is not very efficient to begin with.

**Edit:** following discussions, adding examples that highlights the added value I can see to memory management with such an interface:

```python
X_np, y_np = my_data_loader()  # let's assume that X_np, y_np are typed `np.int64`...

# as a consequence, here the cupy array are also typed `int64`
X_cu = xp.asarray(X_np)
y_cu = xp.asarray(y_np)
X_cu.device

with config_context(array_api_dispatch=True):
    lda = LinearDiscriminantAnalysis()
    # But `lda` requires `floats` to work
    # (see https://github.com/scikit-learn/scikit-learn/blob/f3f51f9b6/sklearn/discriminant_analysis.py#L551)
    # so implicitly, an additional copy of X_cu and y_cu in `float` is going to be created
    # resulting on twice the memory allocation for data.
    X_trans = lda.fit_transform(X_cu, y_cu)
```

for the unwary user, this workflow is safer:

```python
X_np, y_np = my_data_loader()  # let's assume that X_np, y_np are typed `np.int64`...

with config_context(array_api_dispatch=True, array_api_namespace=xp):
    lda = LinearDiscriminantAnalysis()
    # The X_np, y_np are yet again implicitly transformed as cupy float arrays. But
    # the intermediate copy that the users created is skipped (or at worst,
    # deleted right after creation), resulting in better memory management.
    X_trans = lda.fit_transform(X_np, y_np)
``` 

### Describe your proposed solution

The `get_namespace` function can query the `config_context` and force the use of the `__array_namespace__` that is set by the config rather than the `__array_namespace__` that attached to the input (if any). (rather simple solution)

user: fcharras
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_24990: state: open
title: MAINT Split `Splitter` into a `BaseSplitter` and a `Splitter` subclass to allow easier inheritance
body: ### Summary
With #24678, we make it easier for the `Criterion` class to be inherited. However, the Splitter class can also leverage this improvement. We should separate the current `Splitter` class into an abstract base class for "any type of splitting" `BaseSplitter` and an abstract supervisededly splitter class that requires `y` labels `Splitter`. By keeping the names the same, this change is mostly backwards-compatible.

Moreover, we should refactor what `criterion.init` does into its two intended functionalities: i) setting data and ii) moving pointers around and updating criterion statistics.

### Proposed improvement
Based on discussion below, we want to preserve the `y` parameter passing chain of `Tree -> Splitter -> Criterion`. With the exception of `splitter.init`, all other functions can and should be part of the base class without any notion of whether or not the splitter is supervised, or unsupervised. In order to achieve this, we need to separate where `y` is passed to the criterion (currently it is done within `node_reset`.

1. Refactor `criterion.init` into two functions for initializing the data and resetting the pointers (https://github.com/scikit-learn/scikit-learn/blob/b728b2e8b192857f3522007328fa66909ed68787/sklearn/tree/_criterion.pyx#L44-L69):
i) `criterion.init(y, sample_weight, weighted_n_samples, samples)`, which initializes the data for criterion.
ii) `criterion.set_pointer(start, end)`, which sets the pointers to the start/end of the samples we consider
2. Refactor `splitter.init` to pass the value of `y` to the criterion via the new `criterion.init` function (https://github.com/scikit-learn/scikit-learn/blob/b728b2e8b192857f3522007328fa66909ed68787/sklearn/tree/_splitter.pyx#L96)
3. Refactor `splitter.node_reset` to call `criterion.set_pointer` instead of `criterion.init`. (https://github.com/scikit-learn/scikit-learn/blob/9c9c8582dff9f4563aa130ef89f155bad0051493/sklearn/tree/_splitter.pyx#L180)
4. Refactor `Splitter` into `BaseSplitter` and `Splitter` class. The `BaseSplitter` will contain all the methods except for `init`. `Splitter` will subclass `BaseSplitter` and require the `init` function.

This makes Splitter easier to subclass and also removes some of the interdependency of parameters between Tree, Splitter and Criterion.

Once the changes are made, one should verify:

1. If `tree` submodule's Cython code still builds (i.e. `make clean` and then `pip install --verbose --no-build-isolation --editable .` should not error out)
2. verify unit tests inside `sklearn/tree` all pass
3. verify that the asv benchmarks do not show a performance regression.

`asv continuous --verbose --split --bench RandomForest upstream/main <new_branch_name>` and then for side-by-side comparison `asv compare main <new_branch_name>`

### Reference

As discussed in #24577 , I wrote up a doc on proposed improvements to the tree submodule that would:
1. make it easier for 3rd party packages to subclass existing sklearn tree code and
7. make it easier for sklearn itself to make improvements to the tree code with many of the modern improvements to trees

cc: @jjerphan 

@jshinm I think this is a good next issue to tackle because it is more involved, but does not require making any backward incompatible changes to the sklearn tree code.

user: adam2392
labels: Refactor: Code refactor
issue_24972: state: open
title: Dirichlet Multinomial Mixture Model
body: ### Describe the workflow you want to enable

Is there an intention to implement the Dirichlet Multinomial Mixture Model with EM algorithm?

Dirichlet Multinomial Mixture Model is a popular clustering model in NLP, Information Retrieval and Bioinformatics. 

### Describe your proposed solution

I have implemented a usable Dirichlet Multinomial Mixture Model with the Base Mixture class in scikit-learn. If it is needed, I can refine the doc and details and pull it to scikit-learn.

https://github.com/Jck-R/pyDIMM

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: Jck-R
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_24967: state: open
title: Transformer for nominal categories, with the goal of improving category support in decision trees
body: I'd like to find out how keen people would be for adding a transformer like this.

### Describe the workflow you want to enable

Improved support for nominal categories in tree based models. Nominal categories are ones with no order to them, for example colour or country (c.f. ordinal categories that are ordered).

### Describe your proposed solution

In #12866 @amueller linked to [Splitting on categorical predictors in random forests](https://peerj.com/articles/6339). This proposes to replace categorical values (red, green, blue, ..) by cleverly computed numerical values (blue -> 0.643, green ->0.123, ...) that allow you to achieve similar/equal performance by using `<` as splitting decision in a tree node compared to performing an exhaustive search of all possible categorical splits.

I would implement this as a transformer that you use together with a random forest or other tree based model in a pipeline.

### Describe alternatives you've considered, if relevant

There have been several PRs attempting to add native category support to decision trees. #12866 is the latest one. My impression is that it would be cool to have this in trees but that it is not easy to do, several people have tried but no PR has landed yet. The "Breimann trick" that is used is also limited in the number of categorical values it supports, where this new idea seems to support unlimited categorical values and multi class classification.

### Additional context

An earlier paper that is cited in "Splitting categorical predictors...": https://link.springer.com/article/10.1023/A:1009869804967

An open question for me is how to measure how good this kind of transformation is compared to the exhaustive split (which is "the best possible" but too expensive to compute to be used in practice). The method proposed by the paper seems simple (famous last words), and they make claims about the performance but it would be nice to know how close this gets us. Trust, but verify ;)
user: betatim
issue_24903: state: open
title: FIX pairwise_distances not working parallel with custom metric (#24896)
body: 
#### Reference Issues/PRs
Fixes #24896.


#### What does this implement/fix? Explain your changes.
The function pairwise_distances in pairwise.py comes with a forced "threading" joblib backend.
This slows down the system if one uses a python callable as distance function, due to the global interpreter lock - in practice there is no parallelism.
In this fix threading is "soft hinted" as the preferred backend, but is not forced if the user requests another one, e.g. with the code
```
with parallel_backend('loky', n_jobs=n_jobs):
    pairwise_distances(X, metric=myCustomDistanceWrittenInPython)
```

This fix was tested by adding corresponding lines to the testing module sklearn/metrics/tests/test_pairwise.py


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: menta78
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_24901: state: open
title: Inconsistent results with HalvingGridSearchCV
body: ### Describe the bug

`best_params_` does not match top `rank_test_scores` in`cv_results_`.

### Steps/Code to Reproduce

Please see https://stackoverflow.com/q/74408418/4996152

### Expected Results

Expect best params to be top ranked result.

### Actual Results

The output of `best_params_` is not within the top 4 `rank_test_scores` in`cv_results_`.

### Versions

```shell
System:
    python: 3.7.15 (default, Oct 12 2022, 19:14:55)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.21.6
        scipy: 1.7.3
       Cython: 0.29.32
       pandas: 1.3.5
   matplotlib: 3.2.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

user: fickas
issue_24896: state: open
title: forced threading joblib backend in pairwise_distances
body: ### Describe the bug

The function pairwise_distances in pairwise.py comes with a forced "threading" joblib backend.
This slows down the system if one uses a python callable as distance function, due to the global interpreter lock - in practice there is no parallelism.

Is there a reason for this choice?

In my opition the backend should be different if the metric is a custom callable, or at least, the user should be able to force a different one.

### Steps/Code to Reproduce

Invoke pairwise_distance with a callable as metric and n_jobs > 1

```
def myCustomDistanceWrittenInPython(p0, p1):
    return myhexoticmetric

with parallel_backend('loky', n_jobs=n_jobs):
    pairwise_distances(X, metric=myCustomDistanceWrittenInPython)
```

### Expected Results

joblib creates n_jobs processes and runs parallel

### Actual Results

the global interpreter lock of python prevents a correct parllelization

### Versions

```shell
sklearn main branch after commit e947074f63c6602ae15cd57b1fa4f6658040cff7 of Fri Nov 11 05:10:31 2022 -0500
```

user: menta78
issue_24878: state: open
title: Building interactive demos for examples 
body: ### Describe the issue linked to the documentation

This is not an issue but rather a good-to-have feature for documentation and this issue is more of a discussion.

It would be nice to have links to or embedding interactive demos made with Gradio inside documentation. 
I've built two apps that you might see how it looks like:
- Anomaly detection: https://huggingface.co/spaces/scikit-learn/anomaly-detection
- Classification: https://huggingface.co/spaces/scikit-learn/classification

We can have below demo (but more official version 😅) embedded in your documentation.
<img width="1155" alt="Ekran Resmi 2022-11-10 15 11 41" src="https://user-images.githubusercontent.com/53175384/201114005-5fd13d40-990a-412d-abe6-fdd01d0f87d9.png">

The current workflow for users is to download python code or run binder. This reduces amount of work in this workflow too!
If you think embedded demos is a bit overkill what we could do is to create these demos and host them on [Spaces](https://huggingface.co/spaces) and put link to Space instead (like it is with Keras examples or kornia examples, you can see them [here](https://huggingface.co/keras-io) and [here](https://huggingface.co/kornia), they're linked from their tutorials on both [Keras.io/examples](https://keras.io/examples/) and Kornia docs) We will also host the Spaces in better hardware (8 vCPU 32 GiB RAM) to make sure it's always running and all good. ✨ 

You can see how it's implemented in Kornia for rst docs [here](https://raw.githubusercontent.com/kornia/kornia/627ebcaaab1547c362908fc9ad9a68c6316c5993/docs/source/filters.rst). It looks like [this](https://kornia.readthedocs.io/en/latest/filters.html#interactive-demo) inside Kornia docs.

<img width="1353" alt="Ekran Resmi 2022-11-10 15 15 08" src="https://user-images.githubusercontent.com/53175384/201114865-d44b6af5-0b85-44d1-b490-0450d8ea05aa.png">

As for Keras, we just put a redirection link inside docs since their document generation is more layered.  
<img width="1309" alt="Ekran Resmi 2022-11-10 15 09 57" src="https://user-images.githubusercontent.com/53175384/201113694-a4c5b804-dce4-4e72-b23e-bbe449bbc150.png">

We definitely want to reduce the core maintainers' workflow so what would be cool is that we (team at [Gradio](https://gradio.app/) & community)  will be developing the demos and maintaining them through GitHub actions such that they will not break every time there's a breaking change in sklearn (this happens rarely, though 🙂)

Hoping to see how we could collaborate!
user: merveenoyan
issue_24864: state: open
title: LogisticRegression coefficients should be stored in column major order to improve sparse inference performance.
body: ### Describe the workflow you want to enable

I want to enable the workflow of training a `LogisticRegression` model on sparse data, with substantial numbers of features and classes, with acceptable performance. Currently, inference for a `LogisticRegression` model slows rather substantially when using sparse features since the transpose of the coefficient matrix must be stored in row-major order to perform the sparse matrix multiply. Changing the ordering of the coefficient array improves the performance dramatically. This is illustrated in the below code snippet:

```python
import scipy
import numpy as np

from sklearn.linear_model import LogisticRegression

X = np.random.rand(10_000, 10_000)
y = np.random.randint(0, 500, size=10_000)
clf = LogisticRegression(max_iter=10).fit(X, y)

%timeit clf.predict_proba(scipy.sparse.rand(1, 10_000))
# 21.9 ms ± 973 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

%prun clf.predict_proba(scipy.sparse.rand(1, 10_000))
#   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
#        2    0.019    0.010    0.019    0.010 {method 'ravel' of 'numpy.ndarray' objects}
#        1    0.003    0.003    0.022    0.022 _compressed.py:493(_mul_multivector)

clf.coef_ = np.asfortranarray(clf.coef_)

%timeit clf.predict_proba(scipy.sparse.rand(1, 10_000))
# 467 µs ± 11 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

%prun clf.predict_proba(scipy.sparse.rand(1, 10_000))
#   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
#        1    0.000    0.000    0.000    0.000 {built-in method scipy.sparse._sparsetools.csr_matvecs}
#        1    0.000    0.000    0.000    0.000 {method 'choice' of 'numpy.random.mtrand.RandomState' objects}
```

Additional information can be found [here](https://stackoverflow.com/questions/74325646/why-doesnt-scikit-learns-logisticregression-classifier-use-column-major-for-co).

### Describe your proposed solution

My proposed solution is to ensure that the coefficient matrix in `LogisticRegression` is converted to column major order with `np.asfortranarray` once training has completed. For sparse input, this will dramatically improve performance, while for dense inference, it will not degrade performance too much, as demonstrated below:

```python
import scipy
import numpy as np

from sklearn.linear_model import LogisticRegression

X = np.random.rand(10_000, 10_000)
y = np.random.randint(0, 500, size=10_000)
clf = LogisticRegression(max_iter=10).fit(X, y)

X_test = scipy.sparse.rand(1, 10_000)
X_dense = X_test.A

%timeit clf.predict_proba(X_test)
# 23.4 ms ± 253 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit clf.predict_proba(X_dense)
# 2.81 ms ± 277 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

clf.coef_ = np.asfortranarray(clf.coef_)

%timeit clf.predict_proba(X_test)
# 201 µs ± 705 ns per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
%timeit clf.predict_proba(X_dense)
# 3.05 ms ± 48.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

### Describe alternatives you've considered, if relevant

Alternatives include simply ensuring the input to inference is always dense, which results in a 10x speedup. However, this is still a ~10x slowdown relative to the column major ordering solution.

### Additional context

_No response_
user: mrussek
issue_24834: state: open
title: Specify the constraints for `metric="precomputed"`
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
No Reference Issues/PRs

#### What does this implement/fix? Explain your changes.
This PR adds a small note to the documentation of `sklearn.neighbors.NearestNeighbors` to emphasize that a precomputed distance matrix should be a proper metric even if it is passed as a graph and the algorithm is set to `brute`. For instance, the following simple example does not work because the [`NearestNeighbors.kneighobrs` implementation](https://github.com/scikit-learn/scikit-learn/blob/f3f51f9b611bf873bd5836748647221480071a87/sklearn/neighbors/_base.py#L670) assumes that every point is necessarily one of its nearest neighbours.
```
from sklearn.neighbors import NearestNeighbors
from scipy.sparse.csgraph import csgraph_from_dense

neigh = NearestNeighbors(n_neighbors=1, metric='precomputed', algorithm='brute')
dist_mat = pd.DataFrame([
    [100.0,   1.0,      3.0],
    [1.0,  100.0,      2.0],
    [3.0,      2.0,  100.0],
])
graph = csgraph_from_dense(dist_mat)
neigh.fit(graph)
print(neigh.kneighbors_graph().indices)
>>> [2 2 0]
```

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: anton-bushuiev
issue_24812: state: open
title: ENH Make GMM initialization more robust and easily user customizeable
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #23195. Fixes #24811. 


#### What does this implement/fix? Explain your changes.
 - Allows passing in a custom callable to initialize [`GaussianMixture`](https://github.com/scikit-learn/scikit-learn/blob/86b8e5b985f0d113772c4a02982844ce2b09b1f1/sklearn/mixture/_gaussian_mixture.py#L457) class (and the parent [`BaseMixture`](https://github.com/scikit-learn/scikit-learn/blob/86b8e5b985f0d113772c4a02982844ce2b09b1f1/sklearn/mixture/_base.py#L44) class), as discussed in
#23195 (also see comment https://github.com/scikit-learn/scikit-learn/issues/23195#issuecomment-1110762851). Adds appropriate tests, input validation.

 - Additionally, allows easily passing in user defined `responsibilities` to exactly initialize a `GaussianMixture` class (#24811), with the new `responsibilities_init` parameter. The public `get_responsibilities` function is now used in the initialization of the `BaseMixture` class, and is tested properly. 

Docs are expanded to describe the new initialization. Tests are added. See the linked issues for more details.

#### Any other comments?
It has been difficult to robustly initialize GMM since the initialization uses hardcoded parameters and the user really only has control over the type of algorithm to use, but not on any specific initialization, or with more robust user-chosen parameters (which are needed when data is sparse or highly structured, for example). Although it was always possible to hack together private functions from the `_gaussian_mixture.py` to properly do this while using the `_base.py` class for reference, and using for example `clusters.KMeans` to do a fit first to use for initialization, this was difficult in practice. 

It required calculating means, weights, and precisions (after first reverse-engineering and calculating responsibilities), which are not trivial, nor explained. Since all of these are calculated from responsibilities initialized from  the `init_params` inside the GMM class anyway, this PR now adds a way to simple pass in the responsibilities instead, and adds a helper function to calculate them from `KMeans`, `kmeans_plusplus`, or custom initializers, in a generic way.

Also removed some unreachable code and increased test coverage, added tests to show that these changes produce the same behavior as the old initializer, harder to test, code, and added some tests which should act as regression tests or edge case tests. Finally, I am not sure what the best place for this `get_responsibilities` helper function is, and whether it should be exposed in the `__init__.py` for the module. 

All suggestions welcome, as this is my first code contribution to scikit-learn. Black, Flake8, and MyPy were run successful on my local machine, and the docs compiled.
<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: emirkmo
issue_24811: state: open
title: Allow directly passing in responsibilities to GMM
body: > I think it's reasonable to extend the initialization possibilities of GMM...
> Another possibility to directly pass initial responsibilities would also be interesting.    
 Based on original comments by @jeremiedbb in https://github.com/scikit-learn/scikit-learn/issues/23195#issuecomment-1110762851_

It would be great if `responsibilities` could be directly passed into GMM models to initialize it, as the current method of passing in `weights_init`, `means_init`, and `precisions_init` requires accessing several private functions hidden inside different modules if users wanted to simply use their own initializations.

This feature requests builds on top of #23195 which keeps track of passing in a callable into `init_params` for initializing GMM (or future/other mixture models).  However, the issue is separate as the `weights_init`, `means_init`, and `precisions_init` parameters are only defined at the GMM class level but passing in `init_params` is defined at the `BaseMixture` class level, and is a separate effort anyway. 

      
user: emirkmo
issue_24810: state: open
title: [MRG] Add probabilistic estimates to HistGradientBoostingRegressor
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

https://github.com/scikit-learn/scikit-learn/issues/23334

#### What does this implement/fix? Explain your changes.
This PR allows to create probabilistic estimates using `HistGradientBoostingRegressor`. It implements the PGBM method ([reference implementation](https://pgbm.readthedocs.io/en/latest/), [paper](https://arxiv.org/abs/2106.01682) from KDD '21, [video ](https://www.youtube.com/watch?v=quqrxuG_NP8) from KDD '21 providing explanation of the method).

Disclaimer: I am one of the authors of this method.

#### What does it do / how does it work? 
PGBM allows to create probabilistic estimates for gradient boosting models as follows:
- In addition to storing the leaf values of a tree during training, we also store the variance of each leaf during training.
- These variances can be added after training, similar to the leaf values (i.e. the means). 
- We can parameterize a distribution using these learned empirical mean and variances, and sample from this distribution in order to get probabilistic estimates from our `HistGradientBoostingRegressor `model.

I've introduced a couple of additional parameters for `HistGradientBoostingRegressor`, and a new method `sample`, that allows to sample from the learned distribution. I've tried to follow the same conventions as also implemented in e.g. [GaussianMixture](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html).

The additional code mainly comes from trying to keep performance regression as low as possible; this PR is fully backwards compatible (e.g. a model trained using a previous version of Scikit-learn will also load in this version).

I've provided an example that details how it works, and compares it to quantile regression models.

#### Why should we want this / what is the benefit compared to what we already have?
- Quantile regression models require to create a separate model for each required quantile. This method allows to use just a single model to create a rich distribution of outputs. This leads to computational savings (one model vs _n_ models for _n_ quantiles)
- We can get the same probabilistic performance as quantile regression models using just a single model (see the example).
- There is no guarantee that quantile regression models, trained separately, have non-crossing quantiles. If you train two models for a 10% and 90% quantile, there is no guarantee that the first model will actually output values that are strictly less than the second model for every input. A single model, with a single parameterized distribution, does have this guarantee.

#### What is the performance impact of this PR?
I ran the `asv_benchmark` and it gave non-significant changes. This is because I've made 'saving the variance' optional, which means without setting this during initialization of `HistGradientBoostingRegressor`, it runs as fast as the current implementation (there is some additional control flow, but this has negligible impact on performance).

If we enable the storing of the variance (by setting `with_variance=True` in the initialization of `HistGradientBoostingRegressor`, the measured performance impact is approximately 60%, i.e. training a model takes about 60% longer than without storing the variance. This is based on models with a moderate number of samples, i.e. 10.000. Beyond 10.000 (100k, 1mln), the performance difference becomes much less than 60% (more like 5-10%).

For those interested, the key performance hit seems to come from the fact that the histogram structure in `histogram.pyx` needs to store 3 additional variables. I've tried to reuse code as much as possible, but also did not want to impact the existing performance of users who do not need this additional functionality, which thus required copying code.

#### Other things we should know?

- All checks with `pytest` pass, and I've added non-regression tests to also verify my code additions.
- Passes `flake8`, `black` and `mypy`.

This is my first PR for scikit-learn; I may have missed something, forgot something, or did something wrong. Apologies! Let me know. Happy to help.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: elephaint
issue_24802: state: open
title: TST use global_random_seed in sklearn/cluster/tests/test_spectral.py
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
#22827

#### What does this implement/fix? Explain your changes.
Use global_random_seed in tests of test_spectral.py

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: irene000
issue_24788: state: open
title: [WIP] first step in fixing minimum 2 required samples for fixing MLPRegressor attribute error
body: 

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes #24713 

#### What does this implement/fix? Explain your changes.
Added an if loop to check 10% of input training data (X) is at least 2 - based on the comment by @glemaitre in the _validate_data function in the BaseEstimator class in sklearn/base.py

If this is the correct approach to design the fix, what needs to be further done:
1. manual selection in case 2 >= len(X) < 10
2. Testing

#### Any other comments?
Any further comments or directions will be appreciated to design a proper fix.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: mohitthakur13
issue_24786: state: open
title: Gaussian process `log_marginal_likelihood()` `theta` parameter: pass as `log(theta)`?
body: ### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/24765

I opened this as a discussion, but it is probably more suited for an issue, as it may lead to a documentation fix.

System Info

```
System:
    python: 3.10.7 (main, Sep  8 2022, 14:34:29) [GCC 12.2.0]
executable: /usr/bin/python3
   machine: Linux-5.19.0-1-amd64-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.1.2
          pip: 22.2
   setuptools: 59.6.0
        numpy: 1.21.5
        scipy: 1.8.1
       Cython: 0.29.32
       pandas: 1.3.5
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
        version: None
    num_threads: 16
```

<div type='discussions-op-text'>

<sup>Originally posted by **elcorto** October 26, 2022</sup>
## Preliminaries

I'm a bit confused by `GaussianProcess{Regressor,Classifier}`'s internal hyperparameter representation and the nature of the `theta` argument of the `log_marginal_likelihood()` method.

The doc strings of these methods say

```
theta : array-like of shape (n_kernel_params,) default=None
    Kernel hyperparameters for which the log-marginal likelihood is
    evaluated. If None, the precomputed log_marginal_likelihood
    of ``self.kernel_.theta`` is returned.
```

However, the GP's internal hyper optimizer code path seems to work with `log(theta)`.

```py
>>> from sklearn.gaussian_process import GaussianProcessRegressor
>>> from sklearn.gaussian_process.kernels import RBF, WhiteKernel

>>> gp=GaussianProcessRegressor(kernel=RBF()+WhiteKernel())
>>> gp.fit(rand(100,3), rand(100))

>>> gp.kernel_
RBF(length_scale=59.8) + WhiteKernel(noise_level=0.0812)

>>> log_theta=gp.kernel_.theta
>>> log_theta
array([ 4.09017207, -2.51094817])

>>> exp(log_theta)
array([59.75017175,  0.08119122])
```


The value of the `kernel_.theta` attribute is actually `log(theta)`. And indeed, the [`theta` getter][getter] returns `log(theta)`, so they are not *stored* in log format but only returned that way.

The getter's docs say

```
Returns the (flattened, log-transformed) non-fixed hyperparameters.

Note that theta are typically the log-transformed values of the
kernel's hyperparameters as this representation of the search space
is more amenable for hyperparameter search, as hyperparameters like
length-scales naturally live on a log-scale.

Returns
-------
theta : ndarray of shape (n_dims,)
    The non-fixed, log-transformed hyperparameters of the kernel
```

OK.

## Question

When calling `log_marginal_likelihood(theta)`, [the code sets `kernel.theta = theta`][call_setter], which calls [the `theta` setter][setter] which does essentially `exp(theta)`. So from that I'd assume that we need to call `log_marginal_likelihood(log(theta))`. Indeed, continuing with above's code

```py
>>> gp.log_marginal_likelihood_value_
-19.58130329439676

>>> gp.log_marginal_likelihood(log_theta)
-19.58130329439676
```

shows that we need to pass `log(theta)` to `log_marginal_likelihood()`. Also when trying to plot `log_marginal_likelihood` on a grid of hyperparameters, I only get what looks like correct results if I pass in `log(theta)`. If that is true, should the documentation of the `log_marginal_likelihood()` methods be adapted accordingly?

Thanks.


[setter]: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/gaussian_process/kernels.py#L290
[getter]: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/gaussian_process/kernels.py#L266
[call_setter]: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/gaussian_process/_gpr.py#L543
</div>
user: elcorto
issue_24780: state: open
title: Graph metaestimators with a builder API
body: ### Describe the workflow you want to enable

I recently added a new project to scikit-learn-contrib called [skdag](https://github.com/scikit-learn-contrib/skdag) , which allows the construction of workflows as DAGs (directed acyclic graphs), wrapped up in the interface of a metaestimator.

I wanted to allow the project more time to see how it gets adopted before suggesting to include it as a new type of metaestimator in sklearn. However I saw that you already had a similar proposition in #16301 so wanted to make you aware that an actual implementation exists that may provide a useful starting point for you.

If there is an interest now or in the future, I'd be happy to help merge or adapt skdag code to fit in with your plans for sklearn.

### Describe your proposed solution

Skdag implements DAGs with a metaestimator interface. This provides a single utility that acts as a drop-in replacement for pipelines, column transformers, feature unions and stacking estimators. It also allows more complex workflows and visualisations of the workflows.

See the project documentation for more details: https://skdag.readthedocs.io/

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: big-o
issue_24757: state: open
title: PCA crashes when fit to large array
body: ### Describe the bug

When trying to fit a `PCA` model to a large dataset with `shape=(30000, 28000)`, the model fits for a bit more than 30 minutes using nearly all cores and then crashes out of python. I've confirmed that a dataset with `shape=(20000, 28000)` succeeds without issue. I watched both in `htop` and found memory usage to never exceed 30% available memory.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn import decomposition

data = np.random.randn(30000, 28000)
pca_model = decomposition.PCA(0.95, whiten=True)
pca_model.fit(data)
```

### Expected Results

The model should be fit without crashing out of python.

### Actual Results

```
free(): invalid size
Aborted (core dumped)
```

### Versions

```shell
System:
    python: 3.8.10 (default, Jun 22 2022, 20:18:18)  [GCC 9.4.0]
executable: /usr/local/bin/python
   machine: Linux-4.15.0-136-generic-x86_64-with-glibc2.29

Python dependencies:
      sklearn: 1.1.2
          pip: 20.2.4
   setuptools: 63.2.0
        numpy: 1.22.3
        scipy: 1.9.0
       Cython: None
       pandas: 1.4.2
   matplotlib: None
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/container-user/.local/lib/python3.8/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 72

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/container-user/.local/lib/python3.8/site-packages/numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 64

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/container-user/.local/lib/python3.8/site-packages/scipy.libs/libopenblasp-r0-9f9f5dbc.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 64
```

user: luca-of-pci
issue_24751: state: open
title: FEA Ensemble selection from Librairies of Models
body: #### Reference Issues/PRs
Associated issue
https://github.com/scikit-learn/scikit-learn/issues/6329

#### What does this implement/fix? Explain your changes.
It implements EnsembleSelection. I added 2 associated unit tests.
The EnsembleSelection is compatible with both Classifier and Regression tasks.

user: PierrickPochelu
issue_24746: state: open
title: [Refactor, Tree] Python tree class for modularity and consistency of BaseEstimator
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Addresses the Python interface issue in #24000 
Also is a follow up to refactoring of Criterion in #24678 

This should be reviewed and merged after #24678 

#### What does this implement/fix? Explain your changes.
This implements a change to the `BaseDecisionTree` class to enable i) consistency with the assumptions of a `BaseEstimator` and ii) modularity for subclassing of the `BaseDecisionTree` class.

consistency:
- `y=None` is now the default in `fit()`, which is in line w/ the assumptions of a `BaseEstimator` that is not supervised/unsupervised yet. This enables someone to subclass `BaseDecisionTree` and define a completely unsupervised tree as described in #24577 

modularity:
- three private functions are introduced: `_set_criterion()`, `_set_splitter()`, and `_set_tree()`. As demonstrated in #22754 any subclass of the Tree and Splitter class is sufficient to build up a tree using the existing TreeBuilder cython class.

#### Any other comments?
n/a

user: adam2392
issue_24732: state: open
title: Improvement for Gaussian NB by rethinking the variance smoothing
body: ### Describe the workflow you want to enable

## Problem background

In [sklearn.naive_bayes.GaussianNB](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html?highlight=gaussian+nb), a genarative probability model is given by 

P(c| **x** ) = P(c) P( **x** |c)/P( **x** ) , where x<sub>i</sub> ~ N(u<sub>i</sub>, v<sub>i</sub>)

However, sometimes the variance for P( x<sub>i</sub> |c) is zero. In other words, with label c, x<sub>i</sub> is a constant value in the dataset.  

## Current solution in `sklearn.naive_bayes.GaussianNB`

It 's seen from the source that 

```python
self.epsilon_ = self.var_smoothing * np.var(X, axis=0).max()
```

and that

```python
self.var_ = np.zeros((n_classes, n_features))
self.var_[:, :] -= self.epsilon_ 
self.var_[i, :] = new_sigma
self.var_[:, :] += self.epsilon_
```


### Describe your proposed solution

## The problem of the current implementation
You can see, the current implementation would add a  portion of the max variance of all the features(without considering the class), which makes the updated variance never being zero (unless all features are constants).  
This is questionable in two cases:

1.  When there is no feature that has a zero variance. 
 Well, the current algs still adds a small variance to each feature. This seems to be fair(all features are considered), but it is actually dangerous. Not all features are preprossed using something like `StdScaler`, which means their variances can differ dramatically. Adding a big value(coming from the feature with largest variance) blindly would damage the probability computation of the feature with small variance greatly. For example, the variance of the biggest feature is 100, while other features have a variance of 1, let var_smoothing be 1, then the updated variance by current implementation would be 200 and 101-s, making the significant wrong probability calculation of other features. 

2. When there is indeed a feature x1 that has a zero variance under some class c1.
 In this case, adding the max variance exaggerated the instability of this feature. You see, the ground-truth variance of x1|c1 may be something like 0.0001, but due to the max-variance feature x2, it is added by 100. which is obviously unreasonable.  

## My implementation 
1. If there is no variable that is zero-variance, do not apply var-smoothing. 
2. If a feature x1 under some class c1 has a zero variance, use the variance of x1 without knowing the class to be the smoothing variance, instead of using the max variance of all features. 
3. If the variance of x1, even without knowing the class, is zero. throw a warning and ignore this feature for decision. 


## Experiment

I tested my implementation on the famous [ling-spam dataset](https://www.kaggle.com/datasets/mandygu/lingspam-dataset)

- model: I use the “bag of words” model to first do a feature extraction, then `sklearn.naive_bayes.GaussianNB` is applied for both the current implementation and my new implementation
- hyperparameter tuning
  -  to make the competition fair, I applied a parameter tuning to both implementations on validation set
  - I used GridSearchCV to search 'var_smoothing' in [1e-13, 1e-11, 1e-9, 1e-7, 1e-5, 1e-3]
  - I also used `geatpy`(a genetic algorithm lib) to optimize the cross validation loss by tuning var_smoothing

- experiment result on test set
  - current implementation
     ```python
     accuracy=0.9615384615384616
     recall=0.9307692307692308
      f1=0.9603174603174605
     ```
    ![image](https://user-images.githubusercontent.com/41530341/197404136-3b6cb6b1-93c0-4b86-ae47-8af897d83af3.png)
  - my implementation
     ```python
     accuracy=0.9807692307692307
     recall=0.9615384615384616
     f1=0.9803921568627451
     ```
     ![image](https://user-images.githubusercontent.com/41530341/197404183-7e687a01-ecd3-4986-9e38-8507c5aa9cb2.png)
     ![image](https://user-images.githubusercontent.com/41530341/197404176-19e82c70-b97a-4122-b5f9-08eef5b89f94.png)




### Describe alternatives you've considered, if relevant

I know that this implentation includes a partial fit algorithm that can further add more samples to train the model furtherly. I've roughly read the report Stanford CS tech report STAN-CS-79-773, but I am still not sure whether my solution would break the formulation of the partial fit problem or whether the current implementation is necessary for doing an accurate partial fit. Currently I think it doesn't matters, but if you find something wrong please give me some insight. 

I will make a PR if my solution is indeed theoretically and experimentally better. 

### Additional context

_No response_
user: 2catycm
issue_24725: state: open
title: `See also` section for `pairwise_distances` and `paired_distances` does not link to referenced documentation.
body: ### Describe the issue linked to the documentation

The current documentation of [pairwise_distances](https://scikit-learn.org/dev/modules/generated/sklearn.metrics.pairwise_distances.html) and [paired_distances](https://scikit-learn.org/dev/modules/generated/sklearn.metrics.pairwise.paired_distances.html) does not contain hyperlinks working in the `See Also` section. 

Is this the intended behavior? 

### Suggest a potential alternative/fix

I'm not actually sure why this would happen as the references seem to be correct. 
Is it because `pairwise_distances` has hyperlink to `paired_distances` and vice-versa is also done? 
user: Kshitij68
issue_24724: state: open
title: verbose_feature_names_out default value
body: ### Describe the workflow you want to enable

So I obviously love the example in https://scikit-learn.org/dev/auto_examples/miscellaneous/plot_set_output.html#sphx-glr-auto-examples-miscellaneous-plot-set-output-py

But it's a bit of a bummer that the pipeline needs so many parameters.
I wonder if we can revisit the default for ``verbose_feature_names_out``. I think we had a lengthy discussion about it, which I remember as follows

- if we do False by default, pipelines might start to fail that haven't failed before
- if we do "auto" by default (only prefix if required to disambiguate), feature names might start changing unexpectedly in a magic way
- if we do True by default, the names might be awkwardly long unnecessarily

So doing "true" seems the best trade-off for production-grade use....

### Describe your proposed solution

maybe this is a good reason to re-consider adding an "interactive" or "unsafe" mode with interactivity-friendly defaults? 

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: amueller
labels: Needs Decision: Requires decision
issue_24713: state: open
title: AttributeError: 'MLPRegressor' object has no attribute '_best_coefs'
body: The following parameters were working fine with another dataset. When I switched to a new dataset, for some reason an `AttributeError` is occurring. Any ideas?

```python
model = MLPRegressor(hidden_layer_sizes=(10), activation='tanh', batch_size=1000,
  learning_rate_init=0.1, max_iter=5000, momentum=.9, solver='adam', early_stopping=True,
  random_state=1, verbose=1)
X = [[0.,0.,0.,0.2173913,0.,0.5,
  0.,0.46666667,0.76351351,0.41140777,0.55555556,0.03361345,
  0.09022556,0.78107607,0.13250518,1.,0.,0.,
  0.,0.,1.],
 [0.80769231,1.,0.,0.69565217,0.20942029,0.5,
  0.,0.13333333,0.,0.,0.88888889,0.95424837,
  0.96491228,0.83766234,0.97584541,0.,0.,0.,
  0.,0.,0.]] 
y = [1.0, 1.0]
model.fit(X, y)
```

Output:
```
...
Iteration 4997, loss = 0.00001391
Validation score: nan
Iteration 4998, loss = 0.00002568
Validation score: nan
Iteration 4999, loss = 0.00001822
Validation score: nan
Iteration 5000, loss = 0.00001470
Validation score: nan

AttributeError: 'MLPRegressor' object has no attribute '_best_coefs'
```

The error occurs here:
https://github.com/scikit-learn/scikit-learn/blob/7c2a58d51f4528827e9bfe9c43d06c5c1716bfb8/sklearn/neural_network/_multilayer_perceptron.py#L687

The error also seems to completely disappear when there are more than around 10-25 training examples.
user: davidshumway
issue_24681: state: open
title: EHN accept l2 metric with ward linkage in agglomeration clustering
body: closes #18531 

We should be able to use the combination `metric="l2"` and `linkage="ward"`.
It is equivalent to `metric="euclidean"`.
user: glemaitre
issue_24675: state: open
title: suggestion to move (copy?) binder button for example to top or right side bar
body: ### Describe the issue linked to the documentation

Firstly, thanks for having binder links for examples!

In the examples such as https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_1_0.html#sphx-glr-auto-examples-release-highlights-plot-release-highlights-1-1-0-py the binder button is as the bottom of the page

<img width="1469" alt="Screen Shot 2022-10-15 at 11 16 30 PM" src="https://user-images.githubusercontent.com/17162724/196016269-ddf3ddd4-7dbe-4825-ac60-f9c538a6c410.png">

It can sometimes take a while to scroll to get to the button. It would great if it could be accessed quicker with the binder link added to the top or to the right side bar.

### Suggest a potential alternative/fix

I would find it beneficial if the button was at the top (as well as the bottom?) or on the right side bar. This would make it quicker for me to run an example. If the example is long then it can take a bit of scrolling to get to the binder button.

I've mocked up some proposals below. I'm think the button under the title ("A demo on Spectal Biclustering algorithm") would look like a binder button badge as the top of a README e.g. the shield.io format https://mybinder.readthedocs.io/en/latest/howto/badges.html

It's also worth noting that the box in the right isn't accessible and I was able to scroll up to access it

I understand I don't want the badge/link to stick out like a sore thumb and not distract people are simple interested in reading though the examples.

![Screen Shot 2022-10-15 at 11 24 46 PM2](https://user-images.githubusercontent.com/17162724/196016750-134556fb-5839-42ff-8fd2-04e80df58752.png)

user: raybellwaves
labels: Needs Decision: Requires decision
issue_24674: state: open
title: FEA Allow string input for pairwise distances
body: #### Reference Issues/PRs
closes #15932
closes #17991
supersede #17991 

#### What does this implement/fix? Explain your changes.
This allows the user to compute pairwise distance using a custom metric for strings.
user: Kshitij68
issue_24656: state: open
title: Improving stratification in StratifiedGroupKFold
body: ### Describe the workflow you want to enable

I think I found two types of issues in StratifiedGroupKFold. First, StratifiedGroupKFold sometimes doesn't give ideally stratified splits when shuffle=True. Second, I think I found a general bug where the code just isn't doing what I think is intended when shuffle=True. My main change is something of a "new feature", but if this should be instead submitted as a bug report, I can do that.

### Improving the stratification

In the below example, StratifiedGroupKFold is working as intended. It gives train/test splits with identical balances of Y. For this example, there are six groups. For four groups, the an average Y = 0.6, but for two groups the average is Y = 0.4. The train/test split is 50%/50% so the ideal split has train Y = 0.533 and test Y = 0.533 (two groups from the Y = 0.6 and one from Y = 0.4).

However, when random_state=0 is changed to random_state=15, it no longer gives perfectly balanced splits. For random_state=15, the first fold has average Y = 0.6 and the second fold has average Y = 0.466. We generally want StratifiedGroupKFold to give the most ideal splits possible. My solution attempts to achieve this. My solution is also meant to just generally improve upon the author's original algorithm.


``` Python
import numpy as np
from sklearn.model_selection import StratifiedGroupKFold

subjects = 6
X = np.random.normal(size=(subjects*5, 1))
Y = [1, 1, 0, 0, 0]*(1)
Y += [1, 1, 0, 0, 1]*(2)
Y += [1, 1, 0, 0, 0]*(1)
Y += [1, 1, 0, 0, 1]*(2)
Y = np.array(Y)

groups = [[i]*5 for i in range(subjects)]
groups = [item for sublist in groups for item in sublist]

skf = StratifiedGroupKFold(n_splits=2, shuffle=True, random_state=0) # shift this to random_state=15

for train_index, test_index in skf.split(X, Y, groups=groups):
    Y_train, Y_test = Y[train_index], Y[test_index]
    Y_train_mean = np.mean(Y_train)
    Y_test_mean = np.mean(Y_test)
    print(f'{Y_train_mean=:.3f} | {Y_test_mean=:.3f}')
```

### Bug fix

I also found a bug in the original code related to shuffle=True. shuffle=true shuffles a matrix, y_counts_per_group. For this matrix, each row corresponded to a group and the columns were counts of how many times that group had a given value of y. So [[1, 2], [3, 1]] means group0 had one Y=0 & two Y=1, whereas group1 had three Y=0 and one Y=1.
This matrix was being shuffled. 

However, I think this creates a problem with another variable, group_inverse, so the shuffling should occur earlier. Group_inverse maps the index of the X or y arrays to a group number. That is, group_inverse[idxs] yields each idx's group. By shuffling y_counts_per_group, this causes the row-numbering for y_counts_per_group to become misaligned" with the mapping of group_inverse (e.g., row 1 in y_counts_per_group no longer corresponds to group 1 in group_inverse). 

I fixed this to do the shuffling earlier so the two are not misaligned. I noticed this issue because I needed to use y_counts_per_group for my solution below.


### Describe your proposed solution

### Improving stratification

I have a greedy algorithm solution. Basically, when non-perfect stratification is detected, I attempt to swap pairs of groups until the stratification is the best that it can be. I do this for all pairs of groups. If doing so improves the stratification, the swap is made and the process is restarted. The process keeps going until no pair of groups can be swapped to improve the stratification.

Notably, the current code already provides an approach that calculates the level of stratification:
``` Python
std_per_class = np.std(y_counts_per_fold / y_cnt.reshape(1, -1), axis=0)
```

I wrote some code to implement my solution:

``` Python
def _optimize_stratification(self, y_counts_per_fold, y_cnt, y_counts_per_group, groups_per_fold):
    groups = np.array(range(y_counts_per_group.shape[0]))
    np.random.shuffle(groups)
    for group0 in groups:
        for fold0, groups_ in groups_per_fold.items():
            if group0 in groups_:
                break
        for group1 in groups:
            if group0 == group1:
                continue
            for fold1, groups_ in groups_per_fold.items():
                if group1 in groups_:
                    break
            std_per_class = np.std(y_counts_per_fold / y_cnt.reshape(1, -1),
                                   axis=0)

            y_counts_per_fold[fold0] -= y_counts_per_group[group0]
            y_counts_per_fold[fold1] += y_counts_per_group[group0]
            y_counts_per_fold[fold1] -= y_counts_per_group[group1]
            y_counts_per_fold[fold0] += y_counts_per_group[group1]

            std_per_class_new = np.std(
                y_counts_per_fold / y_cnt.reshape(1, -1), axis=0)
            if np.mean(std_per_class_new) < np.mean(std_per_class):
                groups_per_fold[fold0].remove(group0)
                groups_per_fold[fold1].remove(group1)
                groups_per_fold[fold1].add(group0)
                groups_per_fold[fold0].add(group1)
                if self._check_stratification_ideal(y_counts_per_fold,
                                                    warn=False):
                    # If the stratification is ideal, then looping stops
                    return
                else:
                    # If the stratification is not, it runs again
                    return self._optimize_stratification(self,
                                                         y_counts_per_fold,
                                                         y_counts_per_group,
                                                         y_cnt,
                                                         groups_per_fold)
            else:
                y_counts_per_fold[fold0] += y_counts_per_group[group0]
                y_counts_per_fold[fold1] -= y_counts_per_group[group0]
                y_counts_per_fold[fold1] += y_counts_per_group[group1]
                y_counts_per_fold[fold0] -= y_counts_per_group[group1]

``` 

Regardless, even without any changes to how the code functions and even aside from pursuing ideal splits, I think a warning would be helpful to say that the stratification isn't perfect (this function is also called by the one above):

``` Python
def _check_stratification_ideal(self, y_counts_per_fold, warn=False):
        if y_counts_per_fold.shape[0] > 1:
            first_fold = y_counts_per_fold[0]
            for fold in y_counts_per_fold[1:]:
                if not np.array_equal(first_fold, fold):
                    if warn:
                        warnings.warn('Folds are not perfectly stratified.',
                                  UserWarning)
                    return False
        return True
```

Then, in `_split`, these lines are added to check if the stratification is ideal and if not, to run the optimization:
``` Python
if not self._check_stratification_ideal(y_counts_per_fold, warn=False):
    self._optimize_stratification(y_counts_per_fold, y_cnt, y_counts_per_group,
                 groups_per_fold)
```

### Shuffling bug fix

As I mentioned above, I needed to shift the shuffling to fix a general bug. I commented out:
``` Python
# if self.shuffle:
#    rng.shuffle(y_counts_per_group)
```

And added earlier (the original code uses group_inv to calculate y_counts_per_groups in the subsequent lines):
``` Python
if self.shuffle:
    group2shuffled = np.array(range(len(groups_unique)))
    rng.shuffle(group2shuffled)
    groups_inv = group2shuffled[groups_inv]
```


### Describe alternatives you've considered, if relevant

I think my greedy algorithm finds the best solution in cases where groups all have equal sizes and where Y = 0 or 1. I'm not sure about cases where the groups have unequal sizes or where Y can be outside 0 or 1. The optimization would still make the stratification better, but I'm not sure if it will always find the best solution.

A downside to my solution is that users may not care about ideal stratification, and this function could waste their time. However, I don't think the time cost is too major relative to the time needed to actually fit models. I guess this just depends on how many groups people are using. It may be problematic if the number of groups is in the thousands. Implementing something like this may require an optional parameter asking users whether they want this optimization and if so, for how many loops they want to do.

### Additional context

Recently, a new cross-validation function was proposed (issue #24247), RepeatedStratifiedGroupKFold, which uses StratifiedGroupKFold. Having shuffling working properly is needed for RepeatedStratifiedGroupKFold.

Most of the historic conversations surrounding Stratified Group K-fold have been around biomedical/psychological applications. This is where I'm coming from. Having perfect stratification is sometimes vital in these cases.

I haven't submitted a pull request, but please let me know if I should.
user: paulcbogdan
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_24642: state: open
title: New algorithm: DP-Means
body: ### Describe the workflow you want to enable

DP-Means is a nonparametric version of K-Means, which is rooted in the Dirichlet Process Mixture Model.
Proposed by[ Kulis and Jordan in 2011](https://arxiv.org/pdf/1111.0352.pdf), it is already a well cited and established algorithm.

Recently, we have published a version of it which scales very good, comparable to K-Means and is equivalent to the original algorithm (https://openreview.net/pdf?id=rnzVBD8jqlq).

Enabling it in scikit-learn would benefit the community, and would be great in my opinion.

### Describe your proposed solution

We have already utilized scikit-learn codebase in order to implement our algorithm efficiently, (https://github.com/BGU-CS-VIL/pdc-dp-means/tree/main/cluster), so I can easily fix this into a pull-request to scikit-learn code base (will need to add the relevant tests and docs as well).

As such, if this feature request is approved, I will work on the relevant pull request in order to add this to the code base.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: dinarior
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_24638: state: open
title: TransformedTargetRegressor is incompatible with GaussianProcessRegressor when using return_std=True
body: ### Describe the bug

`TransformedTargetRegressor` is incompatible with `GaussianProcessRegressor` when using `return_std=True` due to the fact that the model returns a tuple rather than an array.



### Steps/Code to Reproduce

```python
import numpy as np

from sklearn.preprocessing import PowerTransformer
from sklearn.compose import TransformedTargetRegressor
from sklearn.gaussian_process import GaussianProcessRegressor

from sklearn.datasets import fetch_openml
from sklearn.utils import shuffle

from sklearn.model_selection import train_test_split
from sklearn.compose import make_column_transformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OrdinalEncoder
from sklearn.compose import make_column_selector

def load_ames_housing():
    df = fetch_openml(name="house_prices", as_frame=True)
    X = df.data
    y = df.target

    features = [
        "YrSold",
        "HeatingQC",
        "Street",
        "YearRemodAdd",
        "Heating",
        "MasVnrType",
        "BsmtUnfSF",
        "Foundation",
        "MasVnrArea",
        "MSSubClass",
        "ExterQual",
        "Condition2",
        "GarageCars",
        "GarageType",
        "OverallQual",
        "TotalBsmtSF",
        "BsmtFinSF1",
        "HouseStyle",
        "MiscFeature",
        "MoSold",
    ]

    X = X[features]
    X, y = shuffle(X, y, random_state=0)

    X = X[:600]
    y = y[:600]
    return X, np.log(y)

X, y = load_ames_housing()

cat_selector = make_column_selector(dtype_include=object)
num_selector = make_column_selector(dtype_include=np.number)

cat_tree_processor = OrdinalEncoder(
    handle_unknown="use_encoded_value", unknown_value=-1
)
num_tree_processor = SimpleImputer(strategy="mean", add_indicator=True)

tree_preprocessor = make_column_transformer(
    (num_tree_processor, num_selector), (cat_tree_processor, cat_selector)
)

X_train, X_test, y_train, y_test = train_test_split(X, y)

X_train = tree_preprocessor.fit_transform(X_train)
X_test = tree_preprocessor.transform(X_test)

gpr = GaussianProcessRegressor(random_state=0)
yj = PowerTransformer("yeo-johnson")
tt_gpr = TransformedTargetRegressor(gpr, transformer=yj)

tt_gpr.fit(X_train, y_train)

tt_gpr.predict(X_test, return_std=True)
```

### Expected Results

Predictions are returned from the transformed regressor. It is potentially unclear how the transformer should apply to the predictions given that custom behavior is likely required for each type of transformer breaking the `func`, `inverse_func` API. However if this is the case it would be good to have a more informative error message/something in the documentation.

### Actual Results

```
AttributeError                            Traceback (most recent call last)
[<ipython-input-3-43ceeb18ba48>](https://localhost:8080/#) in <module>
     31 tt_gpr.fit(X_train, y_train)
     32 
---> 33 tt_gpr.predict(X_test, return_std=True)

[/usr/local/lib/python3.7/dist-packages/sklearn/compose/_target.py](https://localhost:8080/#) in predict(self, X, **predict_params)
    273         check_is_fitted(self)
    274         pred = self.regressor_.predict(X, **predict_params)
--> 275         if pred.ndim == 1:
    276             pred_trans = self.transformer_.inverse_transform(pred.reshape(-1, 1))
    277         else:

AttributeError: 'tuple' object has no attribute 'ndim'
```

### Versions

```shell
System:
    python: 3.7.14 (default, Sep  8 2022, 00:06:44)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.21.6
        scipy: 1.7.3
       Cython: 0.29.32
       pandas: 1.3.5
   matplotlib: 3.2.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```
```

user: CompRhys
issue_24634: state: open
title: HalvingRandomSearchCV error
body: ### Describe the bug

https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/model_selection/_search_successive_halving.py#L220
when resources is the hyperparameter of estimator, the best performance is not always at its maximum, right?. so in my opinion, in this case, we should select the best in whole iter instead of the best in last iter.

### Steps/Code to Reproduce

```py
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import  HalvingRandomSearchCV, KFold

X, y = load_iris(return_X_y=True)
clf = RandomForestClassifier(n_estimators=10, random_state=0)  

kf = KFold(shuffle=False)
param_distributions = {"max_depth": [2, 3, 4], "min_samples_split": list(range(2, 12))}
search = HalvingRandomSearchCV(clf, param_distributions, min_resources=15, max_resources=150,factor=3,
                            random_state=0, n_jobs=1,cv=kf, **resource='n_estimators'**
                            ).fit(X, y)

print(search.best_estimator_)
```

### Expected Results

```
RandomForestClassifier(max_depth=3, min_samples_split=5, n_estimators=15,
                       random_state=0)
```

### Actual Results

```
RandomForestClassifier(max_depth=4, min_samples_split=4, **n_estimators=135**,random_state=0)
```

### Versions

```shell
System:
    python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:59:45) [MSC v.1929 64 bit (AMD64)]
executable: C:\software\anaconda\python.exe
   machine: Windows-10-10.0.19043-SP0

Python dependencies:
          pip: 22.0.4
   setuptools: 62.1.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.8.0
       Cython: 0.29.28
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0
```

user: poetair
issue_24616: state: open
title: TST use global_random_seed in sklearn/mixture/tests/test_gaussian_mixture.py
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Towards #22827

#### What does this implement/fix? Explain your changes.
- Added global_random_seed in applicable tests of test_gaussian_mixture.

#### Any other comments?
- Some tests were only tested for 10 seeds for the range 0-9 since all 100 seeds take too long.
- For some of the tests the training samples or features were increased to ensure that the tests pass for the different seeds.
- Some tests are already using seed through parametrization. Do we need to use global_random_seed with them? Currently I left them as they are.
- test_gaussian_mixture_fit takes quite long even with only 10 seeds so I left it.
- test_gaussian_mixture_fit_best_params takes about 0.95 s with 10 seeds and gives two errors. So if I try to increase the training set size or change parameters it goes beyond 1s so I did not modify it.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: OmarManzoor
issue_24615: state: open
title: Inconsistent Results For Logistic Regressions across multiple computers
body: ### Describe the bug

Hey all - 

I'm working on teaching some students some logistic regressions and noticed different computers can produce slightly different intercepts/coefs. At first I thought it was maybe environment differences, but I have been able to reproduce the variances when accounting for the various packages/interpreters. 

I can create a new environment on computer 1 (windows 10, intel 8th gen CPU), computer 2 (windows 10, intel 11th gen CPU), and a coworker's M1 MacBook - all three produce different results. I thought maybe minute differences in Numpy's Openblas or MKL could be the culprit but that yielded the same varying results. I've tried not splitting the data, different random states, different C values, different solvers... 

For the purposes of reproducibility, I ran this is exact code & uploaded the data to Google sheets:

```python
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

bank_df = pd.read_csv('https://docs.google.com/spreadsheets/d/e/2PACX-1vSGfjG4mq1_4HS4iwRN7EZK6YHzPDi8HpB_giY7kiqbDZRsRNjbfhuQ2J6xkHGk1YVYN9H0TxOf2tgw/pub?gid=1909291157&single=true&output=csv')
bank_df.drop(columns=['ID', 'ZIP_Code'], inplace=True)

bank_df['Education'] = bank_df['Education'].astype('category')
new_categories = {1: 'Undergrad', 2: 'Graduate', 3: 'Advanced/Professional'}
bank_df.Education.cat.rename_categories(new_categories, inplace=True)
bank_df = pd.get_dummies(bank_df, prefix_sep='_', drop_first=True)

y = bank_df['Personal_Loan']
X = bank_df.drop(columns=['Personal_Loan'])

# partition data
train_X, valid_X, train_y, valid_y = train_test_split(X, y, test_size=0.4, random_state=1)

# fit a logistic regression (set penalty=l2 and C=1e42 to avoid regularization)
logit_reg = LogisticRegression(penalty="l2", C=1e42, solver='liblinear', random_state=0)
logit_reg.fit(train_X, train_y)
```

All versions of all other packages are held constant. I received this results across the three computers: 


| values | Computer 1 with Sklearn 1.1.2 | Computer 2 with Sklearn 1.1.2 | M1 Macbook with Sklearn 1.1.1 |
|---|---|---|---|
|Intercept|	-12.4919578542609|	-12.6123809362746	| -12.4934360611768|
|Age coef|	-0.037738|	-0.032577	|-0.037685|
|Experience coef|	0.039255|	0.034171|0.039202|
|Income coef|	0.058843|	0.058837	|0.058844|
|Family coef|	0.612243|	0.613192	|0.612251|

I have tried many other things that all produce slightly different results that are not captured in the above table. Multicollinearity is present within the data, but I would still expect consistent results.

I understand the results are effectively the same and this produces little real world impacts. However, when a student is comparing results and they see 12.4 vs 12.6 which is a decently large difference. I think some of these differences are too large to just be contributed to floating-point computations -- especially for size of the data. I assume I'm missing something, but I'm reaching the end of my rope on troubleshooting. Thoughts / expected behavior? 

### Steps/Code to Reproduce

```python
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression

bank_df = pd.read_csv('https://docs.google.com/spreadsheets/d/e/2PACX-1vSGfjG4mq1_4HS4iwRN7EZK6YHzPDi8HpB_giY7kiqbDZRsRNjbfhuQ2J6xkHGk1YVYN9H0TxOf2tgw/pub?gid=1909291157&single=true&output=csv')
bank_df.drop(columns=['ID', 'ZIP_Code'], inplace=True)

bank_df['Education'] = bank_df['Education'].astype('category')
new_categories = {1: 'Undergrad', 2: 'Graduate', 3: 'Advanced/Professional'}
bank_df.Education.cat.rename_categories(new_categories, inplace=True)
bank_df = pd.get_dummies(bank_df, prefix_sep='_', drop_first=True)

y = bank_df['Personal_Loan']
X = bank_df.drop(columns=['Personal_Loan'])

# partition data
train_X, valid_X, train_y, valid_y = train_test_split(X, y, test_size=0.4, random_state=1)

# fit a logistic regression (set penalty=l2 and C=1e42 to avoid regularization)
logit_reg = LogisticRegression(penalty="l2", C=1e42, solver='liblinear')
logit_reg.fit(train_X, train_y)
```

### Expected Results

| values | Computer 1 with Sklearn 1.1.2 | Computer 2 with Sklearn 1.1.2 | M1 Macbook with Sklearn 1.1.1 |
|---|---|---|---|
|Intercept|	-12.4919578542609|	-12.6123809362746	| -12.4934360611768|
|Age coef|	-0.037738|	-0.032577	|-0.037685|
|Experience coef|	0.039255|	0.034171|0.039202|
|Income coef|	0.058843|	0.058837	|0.058844|
|Family coef|	0.612243|	0.613192	|0.612251|

### Actual Results

n/a

### Versions

```shell
System:
    python: 3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\Nick\.conda\envs\sktest\python.exe
   machine: Windows-10-10.0.19044-SP0

Python dependencies:
      sklearn: 1.1.2
          pip: 22.2.2
   setuptools: 63.4.1
        numpy: 1.23.1
        scipy: 1.9.1
       Cython: None
       pandas: 1.4.4
   matplotlib: None
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       filepath: C:\Users\Nick\.conda\envs\sktest\Lib\site-packages\sklearn\.libs\vcomp140.dll
         prefix: vcomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 12

       filepath: C:\Users\Nick\.conda\envs\sktest\Library\bin\mkl_rt.1.dll
         prefix: mkl_rt
       user_api: blas
   internal_api: mkl
        version: 2021.4-Product
    num_threads: 6
threading_layer: intel
```

user: NickBrecht
labels: Needs Investigation: Issue requires investigation
issue_24614: state: open
title: GLM doesn't have an offset option
body: ### Describe the workflow you want to enable

As described at #24155, GLM should also support the offset option.

### Describe your proposed solution

For example
```python
from sklearn.linear_model import PoissonRegressor

glm = PoissonRegressor(
                        alpha=0,
                        fit_intercept=False, 
                        max_iter=300,
                        offset=A # offset option
                      )
glm.fit(X_train, y_train)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: m7142yosuke
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_24601: state: open
title: Add tolerance `tol` to LinearRegression for sparse matrix solver lsqr
body: ===BEGIN EDIT===
## Description
`LinearRegression` should have a parameter `tol` that is passed to the LSQR routine for solving with sparse matrices. This way, it should be (more or less) equal to `Ridge(alpha=0, sol="lsqr", tol=..)`
===END EDIT===

### Description

`linear_model.LinearRegression` performs different on sparse matrix than on numpy arrays. The built-in unit test `test_linear_regression_sparse_equal_dense` works well with two features, but not with other feature counts, e.g. `n_features=14`. Other combinations of `n_sample` and `n_features` lead to even higher discrepancies.

There was a similar issue (https://github.com/scikit-learn/scikit-learn/issues/13460) in 2019, that was fixed (https://github.com/scikit-learn/scikit-learn/pull/13279), and complemented by mentioned unit test.

### Steps/Code to Reproduce

Original code from https://github.com/scikit-learn/scikit-learn/blob/c674e589f9aa19ebd1151c19413622f96c8ed368/sklearn/linear_model/tests/test_base.py#L213

Only modification: `n_features = 14`.

```python
import pytest
import numpy as np
from scipy import sparse
from sklearn.utils._testing import assert_allclose
from sklearn.linear_model import LinearRegression

def test_linear_regression_sparse_equal_dense(normalize, fit_intercept):
    # Test that linear regression agrees between sparse and dense
    rng = np.random.RandomState(0)
    n_samples = 200
    n_features = 14
    X = rng.randn(n_samples, n_features)
    X[X < 0.1] = 0.0
    Xcsr = sparse.csr_matrix(X)
    y = rng.rand(n_samples)
    params = dict(normalize=normalize, fit_intercept=fit_intercept)
    clf_dense = LinearRegression(**params)
    clf_sparse = LinearRegression(**params)
    clf_dense.fit(X, y)
    clf_sparse.fit(Xcsr, y)
    assert clf_dense.intercept_ == pytest.approx(clf_sparse.intercept_)
    assert_allclose(clf_dense.coef_, clf_sparse.coef_)
    
test_linear_regression_sparse_equal_dense(False, False)
```

### Expected Results

Coefficients from both regressions should be equal. The test shouldn't throw any error.

### Actual Results

Test throws an AssertionError as follows:

```python
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-10-52fe7d4efb39> in <module>
     23     assert_allclose(clf_dense.coef_, clf_sparse.coef_)
     24 
---> 25 test_linear_regression_sparse_equal_dense(False, False)

<ipython-input-10-52fe7d4efb39> in test_linear_regression_sparse_equal_dense(normalize, fit_intercept)
     21     clf_sparse.fit(Xcsr, y)
     22     assert clf_dense.intercept_ == pytest.approx(clf_sparse.intercept_)
---> 23     assert_allclose(clf_dense.coef_, clf_sparse.coef_)
     24 
     25 test_linear_regression_sparse_equal_dense(False, False)

    [... skipping hidden 2 frame]

AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 1 / 14 (7.14%)
Max absolute difference: 3.29286053e-09
Max relative difference: 2.61247094e-07
 x: array([ 0.119148,  0.138634,  0.045699,  0.09633 ,  0.095532,  0.052054,
        0.048282,  0.06737 ,  0.074815, -0.003684,  0.083319,  0.064834,
        0.124406,  0.070745])
 y: array([ 0.119148,  0.138634,  0.045699,  0.09633 ,  0.095532,  0.052054,
        0.048282,  0.06737 ,  0.074815, -0.003684,  0.083319,  0.064834,
        0.124406,  0.070745])
```

### Versions

```shell
System:
    python: 3.6.8 (default, Aug 13 2020, 07:46:32)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
executable: /home/envs/my_env/bin/python3
   machine: Linux-3.10.0-1160.62.1.el7.x86_64-x86_64-with-redhat-7.9-Maipo

Python dependencies:
          pip: 21.3.1
   setuptools: 39.2.0
      sklearn: 0.24.2
        numpy: 1.19.5
        scipy: 1.5.4
       Cython: None
       pandas: 1.1.5
   matplotlib: 3.3.4
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

user: epplef
issue_24577: state: open
title: [RFC] Modularize 'Criterion' class inside tree submodule to allow 3rd party extensions for data-based criterion
body: ### Summary

There exists other criterions, where additional arguments would need to be passed in. The current `Criterion` class is not actually a generic Criterion class since it explicitly assumes that the Criterion is supervised-learning based and does NOT require the data `X` itself. The `init(...)` function explicitly passes in `y`.

I am trying to implement and subclass `Criterion` to implement Unsupervised Random Forests (https://arxiv.org/abs/1907.02844), which use the data within the Criterion. I would like to subclass the `Criterion` class for my own `UnsupervisedCriterion`. However, the `init` function assumes access to a `y` label, but everything else about it is fine and correct.

Instead I would like to define my own `init` function, but keep the rest of the Criterion API. This way, I can define an unsupervised criterion with the `init` function:

```python
    cdef int init(self, const DTYPE_T[:, ::1] X, DOUBLE_t* sample_weight,
                  double weighted_n_samples, SIZE_t* samples, SIZE_t start,
                  SIZE_t end) nogil except -1
```

and I could then define the rest of the functions, while making sure I am in strict compliance with sklearn's Tree submodule.

### Proposed Solution

Add a class called `BaseCriterion` and migrate all functions/properties of the current `Criterion` class, EXCEPT for `init` function to `BaseCriterion`.

This is completely backwards-compatable and ensures 3rd party API can leverage the existing API contract for Criterions using the `BaseCriterion` class.

### Alternatives

Alternatively, I can do what `scikit-survival` does and pass in this additional data to `__cinit__`, but this is weird because you basically need to pass in data to a Criterion class before it's used with Tree/Splitter. Rn `init` is called within the Tree/Splitter, which ensures the `y` that `Tree/Splitter` have access to is the same as `Criterion`. If you are forced to pass things through `__cinit__` it is quite possible to introduce runtime bugs because your data changes after you have already instantiated a criterion class.

### Alternatives2

Another possibility that I'm not 100% sure on is that `Criterion.init(...)` is passed `y` and just resets it to the passed in `y` data. However, it never modifies `y`. Instead, it should only init the 'y' variable once.... Perhaps something like `Criterion.init_data(y)`, which separates initialization of the data compared to initialization of different variables.
user: adam2392
labels: Refactor: Code refactor
issue_24540: state: open
title: Exit Code -1073741819 when doing K-means++ clustering
body: ### Describe the bug

Unfortunately I am getting an exit code in Pycharm when doing clustering with k-means++.
I tried nearly everything. Setup new Pycharm project try using different versions of numpy or sklearn.

### Steps/Code to Reproduce

```python
def __cluster(num_clusters: int, data: list[list[float]]):
    km = KMeans(n_clusters=num_clusters)
    return km.fit_predict(data)


distributions_filename: str = "distributions.json"
with open(distributions_filename) as f:
    distributions: list[list[float]] = json.load(f)
    buckets = __cluster(num_clusters=200, data=distributions)
```
### Expected Results

clustered buckets

### Actual Results

before the clustering is finished I get an error: "Process finished with exit code -1073741819 (0xC0000005)"

### Versions

```shell
System:
    python: 3.10.7 (tags/v3.10.7:6cc6b13, Sep  5 2022, 14:08:36) [MSC v.1933 64 bit (AMD64)]
executable: C:\Users\maron\Desktop\test\venv\Scripts\python.exe
   machine: Windows-10-10.0.19044-SP0

Python dependencies:
      sklearn: 1.1.2
          pip: 21.3.1
   setuptools: 60.2.0
        numpy: 1.23.3
        scipy: 1.9.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: C:\Users\maron\Desktop\test\venv\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 32

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\maron\Desktop\test\venv\Lib\site-packages\numpy\.libs\libopenblas.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.gfortran-win_amd64.dll
        version: 0.3.20
threading_layer: pthreads
   architecture: Zen
    num_threads: 24

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\maron\Desktop\test\venv\Lib\site-packages\scipy\.libs\libopenblas.PZA5WNOTOH6FZLB2KBVKAURAKVTFSNNU.gfortran-win_amd64.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: Zen
    num_threads: 24
```

user: maronjan
labels: Needs Investigation: Issue requires investigation
issue_24525: state: open
title: Should we continue to support compiler=intelem?
body: I have an build refactor removing `distutils` and `numpy.disutils` and only uses `setuptools` that successfully builds our wheels and passes tests. I think it is best to move to a pure `setuptools` implementation first, because there are still some lingering issues `meson`. For example, no editable wheels with `meson`: https://github.com/FFY00/meson-python/issues/47.

`setuptools` does not make it easy to support custom compilers: https://github.com/pypa/setuptools/issues/2806. It is doable, but it requires more complexity. (I have not got the intelem compiler fully working on our CI yet, but it is working locally in Docker).

Furthermore, I think our [Intel ICC job](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=47113&view=logs&j=8628a494-79d0-53fa-274c-1b00464f7121&t=044d955c-661f-5416-d272-56c1ab35449e) is not checking the Intel build correctly. Specifically, the following compiles the code correctly with `icc`:

https://github.com/scikit-learn/scikit-learn/blob/681ab94222a9ba5f7b39f768f0ab92873905541a/build_tools/azure/install.sh#L129

But a few lines later, we run:

https://github.com/scikit-learn/scikit-learn/blob/681ab94222a9ba5f7b39f768f0ab92873905541a/build_tools/azure/install.sh#L141

which overrides the icc compiled extensions with gcc compiled extensions. One can see that gcc is used in the [build logs](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=47113&view=logs&j=8628a494-79d0-53fa-274c-1b00464f7121&t=044d955c-661f-5416-d272-56c1ab35449e).

Also in the logs we see that `icc` itself is going to be deprecated:

```
icc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message.
```

Should we support continue to support and build with the Intel compiler?

CC @jeremiedbb @ogrisel 
user: thomasjpfan
issue_24502: state: open
title: RFC Should pairwise_distances preserve float32 ?
body: Currently the dtype of the distance matrix returned by `pairwise_distances` is not very consistent, depending on the metric and on the value of n_jobs.

For float64 input, everything is consistent: the returned matrix is always in float64.
For mixed float64 X and float32 Y, the return matrix is also always in float64 and this is what should be expected imo.

The troubles come when both X and Y are float32.
- for sklearn metrics:
  - `euclidean` and `cosine`: result is always float32
  - `manhattan`: result is float64 if n_jobs=1 and float32 otherwise
- for scipy metrics: result is float64 if n_jobs=1 and float32 otherwise
  Note that scipy cdist/pdist always returns float64.

Hence the question: should `pairwise_distances` preserve float32 ?

My opinion is that it should since `pairwise_distances` can be used as an intermediate step during fit and since there's ongoing work towards preserving float32 in estimators (see https://github.com/scikit-learn/scikit-learn/issues/11000 for transfromers for instance).

An argument against that could be reducing the numerical instabilities. A potential solution could be to use float64 accumulators for the intermediate computations only, still returning a float32 dist matrix. Note that with https://github.com/scikit-learn/scikit-learn/pull/23958 we might not need to use the scipy metrics anymore, in favor of the ones defined in `dist_metrics`, and using float64 accumulators would be easier to implement generally.

Answering this question will help to not go in the wrong direction in https://github.com/scikit-learn/scikit-learn/pull/23958
user: jeremiedbb
issue_24501: state: open
title: plot_learning_curve.py should not sort the fit time axis before plotting
body: Dears,

About 10 months ago, the `plot_learning_curve.py` example was changed by Mr. @thomasjpfan to sort the `fit_time` plot axis.

In my humble opinion, that's wrong because a learning curve is train-size ascending regardless the time it spent training.  For very small train sizes/intervals one can se a slightly smaller train size taking a slightly longer time to fit.

Furthermore, a computer performance may oscillates in any learning curve step which should not affect the actual order of the learning curve steps in the plot.

https://github.com/scikit-learn/scikit-learn/blob/ae6bf39b310ed5bb46349c831a05f55bba921dcc/examples/model_selection/plot_learning_curve.py#L168
user: mauriciocramos
issue_24491: state: open
title: Weekly CI run with NVidia GPU hardware
body: Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)
user: ogrisel
issue_24486: state: open
title: GroupShuffleSplit chokes on pd.Int16Dtype() with a cryptic error
body: ### Describe the bug

`GroupShuffleSplit` chokes on `pd.Int16Dtype()` with a cryptic error.
It looks like internally the data series gets converted to a list, and list comparison returns a scalar, while an iterable is expected

### Steps/Code to Reproduce

```python
data = pd.DataFrame({"clusters": [1, 2, 3, pd.NA, pd.NA],
                    "x" : [0,1,2,3,4]} )

splitter = GroupShuffleSplit(test_size=.2, n_splits=2, random_state = 7)
split = splitter.split(data, groups=data['clusters'])
train_inds, test_inds = next(split)
```

### Expected Results

e.g. `[0,1,3]` and `[2,4]`

### Actual Results

```
AttributeError                            Traceback (most recent call last)
Input In [23], in <cell line: 3>()
      1 splitter = GroupShuffleSplit(test_size=.2, n_splits=2, random_state = 7)
      2 split = splitter.split(data, groups=data['clusters'])
----> 3 train_inds, test_inds = next(split)

File /opt/conda/lib/python3.8/site-packages/sklearn/model_selection/_split.py:1600, in BaseShuffleSplit.split(self, X, y, groups)
   1570 """Generate indices to split data into training and test set.
   1571 
   1572 Parameters
   (...)
   1597 to an integer.
   1598 """
   1599 X, y, groups = indexable(X, y, groups)
-> 1600 for train, test in self._iter_indices(X, y, groups):
   1601     yield train, test

File /opt/conda/lib/python3.8/site-packages/sklearn/model_selection/_split.py:1805, in GroupShuffleSplit._iter_indices(self, X, y, groups)
   1803 if groups is None:
   1804     raise ValueError("The 'groups' parameter should not be None.")
-> 1805 groups = check_array(groups, ensure_2d=False, dtype=None)
   1806 classes, group_indices = np.unique(groups, return_inverse=True)
   1807 for group_train, group_test in super()._iter_indices(X=classes):
   1808     # these are the indices of classes in the partition
   1809     # invert them into data indices

File /opt/conda/lib/python3.8/site-packages/sklearn/utils/validation.py:805, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
    799         raise ValueError(
    800             "Found array with dim %d. %s expected <= 2."
    801             % (array.ndim, estimator_name)
    802         )
    804     if force_all_finite:
--> 805         _assert_all_finite(array, allow_nan=force_all_finite == "allow-nan")
    807 if ensure_min_samples > 0:
    808     n_samples = _num_samples(array)

File /opt/conda/lib/python3.8/site-packages/sklearn/utils/validation.py:126, in _assert_all_finite(X, allow_nan, msg_dtype)
    122     raise ValueError("Input contains NaN")

AttributeError: 'bool' object has no attribute 'any'
```

### Versions

```shell
System:
    python: 3.8.8 (default, Apr 13 2021, 19:58:26)  [GCC 7.3.0]
executable: /opt/conda/bin/python
   machine: Linux-5.4.162-86.275.amzn2.x86_64-x86_64-with-glibc2.10

Python dependencies:
          pip: 22.1.2
   setuptools: 59.5.0
      sklearn: 1.0.2
        numpy: 1.22.3
        scipy: 1.6.3
       Cython: 0.29.23
       pandas: 1.2.4
   matplotlib: 3.4.2
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```

user: DSLituiev
issue_24415: state: open
title: [WIP] Implement PCA on sparse noncentered data
body: Will fix #12794 when complete.

## TODOs
- [ ] test on large random data
  - [ ] $min(m, n)$ at least a few hundred
  - [ ] parametrize on `whiten`
  - [ ] use `global_random_seed`
    - [ ] test locally using `SKLEARN_TESTS_GLOBAL_RANDOM_SEED="all" pytest sklearn/decomposition/tests/test_pca.py::test_pca_sparse`
- [ ] test `.transform`
  - [ ] on random inputs
  - [ ] sparse model on dense inputs
  - [ ] dense model on sparse inputs
- [ ] make a changelog entry
- [ ] implement support for LOBPCG
- [ ] implement support for PROPACK
user: andportnoy
issue_24411: state: open
title: MLPRegressor - Validation score wrongly defined
body: ### Describe the bug

In MLPRegressor, if the option early_stopping is set as True, the model will monitor the loss calculated on the validation set in stead of the training set, using the same loss formulation which is the mean squared error. However, as implemented in the line 719 in the source code:
```
self.validation_scores_.append(self.score(X_val, y_val))
```
The function "score", which returns (to confirm) the coefficient of determination, is used.  This is not correct. It should be something like:
```
self.validation_scores_.append(mean_squared_error(self.predict(X_val), y_val))
```


### Steps/Code to Reproduce

Sorry, I don't have time to write a simple code. But the error is quite clear.

### Expected Results

The validation score must be mean squared error.

### Actual Results

Coefficient of determination

### Versions

```shell
1.1.1
```

user: ducvinh-nguyen
issue_24409: state: open
title: GridSearchCV does not seem to recognize whether estimators from StackingClassifier are fitted or not
body: ### Describe the bug

There seems to be a bug with the combination of `GridSearchCV` and `StackingClassifier` when the parameter `cv` of  `StackingClassifier` is set to 'prefit'. With this option, the estimators of the `StackingClassifier` should be fitted before fitting the stacked model, and only the final_estimator would then be fitted. When including the `StackingClassifier` within `GridSearchCV` however, the fact that estimators have already been fitted does not seem to be recognized.



### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import GradientBoostingClassifier, StackingClassifier
from sklearn.model_selection import GridSearchCV

# Creating toy data set
n_features = 3
n_instances = 40
train = np.random.rand(n_instances, n_features)
label = np.random.randint(0,2,n_instances)

# Declaring estimators
log_clf = LogisticRegression()
gau_clf = GaussianNB()

# Fitting estimators
log_clf.fit(train, label)
gau_clf.fit(train, label)

# Creating stacked model
estimators = [
    ("log", log_clf),
    ("gau", gau_clf)
]

boost = GradientBoostingClassifier()
stack = StackingClassifier(estimators=estimators,
                          final_estimator=boost,
                          cv = 'prefit')

# Creating the Grid CV
param_search = {
    'final_estimator__max_depth': [1]
    }

gridcv = GridSearchCV(stack, 
                      param_grid=param_search)

# Fitting the stack and gridcv models
stack.fit(train, label) # works fine 
gridcv.fit(train, label) # sklearn.exceptions.NotFittedError: This LogisticRegression instance is not fitted yet. 
                         # Call 'fit' with appropriate arguments before using this estimator

```

### Expected Results

In the above code, the `stack` model works fine and no error related to the estimators' previous fitting is thrown.

However, when included in the GridSearchCV, the fact that estimator models have been fitted already does not seem to be recognized and a `sklearn.exceptions.NotFittedError` error is prompted. 



### Actual Results

```
Traceback (most recent call last):
  File "c:\Users\levesque\Documents\Python\AMEX\errorExample.py", line 39, in <module>
    gridcv.fit(train, label) # sklearn.exceptions.NotFittedError: This LogisticRegression instance is not fitted yet. Call 'fit' with appropriate
arguments before using this estimator
  File "C:\Users\levesque\Documents\Python\AMEX\env\lib\site-packages\sklearn\model_selection\_search.py", line 875, in fit
    self._run_search(evaluate_candidates)
  File "C:\Users\levesque\Documents\Python\AMEX\env\lib\site-packages\sklearn\model_selection\_search.py", line 1379, in _run_search
    evaluate_candidates(ParameterGrid(self.param_grid))
  File "C:\Users\levesque\Documents\Python\AMEX\env\lib\site-packages\sklearn\model_selection\_search.py", line 852, in evaluate_candidates
    _warn_or_raise_about_fit_failures(out, self.error_score)
  File "C:\Users\levesque\Documents\Python\AMEX\env\lib\site-packages\sklearn\model_selection\_validation.py", line 367, in _warn_or_raise_about_fit_failures
    raise ValueError(all_fits_failed_message)
ValueError:
All the 5 fits failed.
It is very likely that your model is misconfigured.
You can try to debug the error by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
5 fits failed with the following error:
Traceback (most recent call last):
  File "C:\Users\levesque\Documents\Python\AMEX\env\lib\site-packages\sklearn\model_selection\_validation.py", line 686, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "C:\Users\levesque\Documents\Python\AMEX\env\lib\site-packages\sklearn\ensemble\_stacking.py", line 584, in fit
    return super().fit(X, self._le.transform(y), sample_weight)
  File "C:\Users\levesque\Documents\Python\AMEX\env\lib\site-packages\sklearn\ensemble\_stacking.py", line 183, in fit
    check_is_fitted(estimator)
  File "C:\Users\levesque\Documents\Python\AMEX\env\lib\site-packages\sklearn\utils\validation.py", line 1345, in check_is_fitted
    raise NotFittedError(msg % {"name": type(estimator).__name__})
sklearn.exceptions.NotFittedError: This LogisticRegression instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.
```

### Versions

```shell
System:
    python: 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)]
executable: C:\Users\levesque\Documents\Python\AMEX\env\Scripts\python.exe
   machine: Windows-10-10.0.17763-SP0

Python dependencies:
      sklearn: 1.1.2
          pip: 22.2.2
   setuptools: 57.4.0
        numpy: 1.23.1
        scipy: 1.9.0
       Cython: None
       pandas: 1.4.3
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: C:\Users\levesque\Documents\Python\AMEX\env\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\levesque\Documents\Python\AMEX\env\Lib\site-packages\numpy\.libs\libopenblas.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.gfortran-win_amd64.dll
        version: 0.3.20
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\levesque\Documents\Python\AMEX\env\Lib\site-packages\scipy\.libs\libopenblas.PZA5WNOTOH6FZLB2KBVKAURAKVTFSNNU.gfortran-win_amd64.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8
```

user: piklev
issue_24381: state: open
title: Inconsistency in AUC ROC and AUPR API
body: ### Describe the bug

When only one class is present on the groundtruth. The function `roc_auc_score` throws an `ValueError` and exits while the `average_precision_score` returns `-0.0`. I feel that both functions should return similar output in this case.

### Steps/Code to Reproduce

```python
from sklearn.metrics import roc_auc_score,  average_precision_score

y_true = [1]*10
prob = [0.49]*10

auc_roc = roc_auc_score(y_true, prob)
aupr= average_precision_score(y_true, prob)

```

### Expected Results

auc_roc = roc_auc_score(y_true, prob) >> throws `ValueError: Only one class present in y_true. ROC AUC score is not defined in that case`

aupr=-0.0

### Actual Results

Both should have similar outputs

### Versions

```shell
Python dependencies:                                                                                                
      sklearn: 1.1.2                                                                                                
          pip: 22.0.2                                                                                               
   setuptools: 59.6.0                                                                                               
        numpy: 1.23.2                                                                                               
        scipy: 1.9.1                                                                                                
       Cython: None                                                                                                 
       pandas: 1.4.3                                                                                                
   matplotlib: 3.5.3                                                                                                
       joblib: 1.1.0                                                                                                
threadpoolctl: 3.1.0  

Built with OpenMP: True                                                                                             
                                                                                                                    
threadpoolctl info:          
       user_api: openmp      
   internal_api: openmp      
         prefix: libgomp     
       filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/sitepackages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0    

version: None        
    num_threads: 24          

       user_api: blas        
   internal_api: openblas    
         prefix: libopenblas                              
       filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-742d56dc.
3.20.so                      
        version: 0.3.20      
threading_layer: pthreads    
   architecture: SkylakeX    
    num_threads: 24          

       user_api: blas        
   internal_api: openblas    
         prefix: libopenblas                              
       filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-9f9f5dbc.3.1
8.so                         
        version: 0.3.18      
threading_layer: pthreads    
   architecture: SkylakeX    
    num_threads: 24          

       user_api: openmp      
   internal_api: openmp      
         prefix: libgomp     
       filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/site-packages/torch/lib/libgomp-a34b3233.so.1        
        version: None        
    num_threads: 12
```

user: a-parida12
issue_24369: state: open
title: DOC Clarify documentation writing guideline
body: ### Describe the issue linked to the documentation

The ['Guidelines for writing documentation'](https://scikit-learn.org/dev/developers/contributing.html#guidelines-for-writing-documentation) section seems to be specifically about docstrings and the suggestions don't seem to be as relevant for other types of documentation, e.g., example, tutorial, usage pages. 

### Suggest a potential alternative/fix

Clarify that this section is about docstrings. Potentially add sections on guidelines for writing examples, usage and tutorial pages?
user: lucyleeow
issue_24355: state: open
title: `type_of_target` returns `unknown` for valid arrays of dtype `object`
body: ### Describe the bug

`sklearn.utils.multiclass.type_of_target` returns unknown for arrays of integers if they have a dtype of `object`, when it should instead return a valid type.

I would be happy to contribute a fix, but I'm not entirely sure how the failing condition should be corrected:
https://github.com/scikit-learn/scikit-learn/blob/45756377c748d84aa52f66950b8d9eeefc31456c/sklearn/utils/multiclass.py#L317-L319

### Steps/Code to Reproduce

```python
import pandas as pd
from sklearn.utils.multiclass import type_of_target

series = pd.Series(['a', 'b'])
series.loc[series == 'a'] = 0
series.loc[series == 'b'] = 1

print(series)
print(type_of_target(series))
print(type_of_target(list(series)))
```
Output is `unknown` whereas once converted to a list, the output is `binary` as expected.

### Expected Results

`type_of_target` returns a valid type (`binary`).

### Actual Results

This issue resulted in enigmatic errors when using dependent functions such as `confusion_matrix`, e.g.:
```
Traceback (most recent call last):
  File "test.py", line 214, in <module>
    cm = confusion_matrix(y_test, prediction_classes)
  File "~/.pyenv/versions/wat/lib/python3.8/site-packages/sklearn/metrics/_classification.py", line 307, in confusion_matrix
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)
  File "~/.pyenv/versions/wat/lib/python3.8/site-packages/sklearn/metrics/_classification.py", line 93, in _check_targets
    raise ValueError(
ValueError: Classification metrics can't handle a mix of unknown and binary targets
```

### Versions

```shell
System:
    python: 3.8.0 (default, Oct  6 2020, 12:20:13)  [GCC 6.5.0 20181026]
executable: ~/.pyenv/versions/wat/bin/python3
   machine: Linux-4.15.0-132-generic-x86_64-with-glibc2.27

Python dependencies:
      sklearn: 1.1.2
          pip: 22.0.4
   setuptools: 58.1.0
        numpy: 1.23.2
        scipy: 1.6.0
       Cython: 0.29.21
       pandas: 1.1.3
   matplotlib: 3.3.3
       joblib: 1.0.0
threadpoolctl: 2.1.0

Built with OpenMP: True
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 237, in 'calling callback function'
  File "~/.pyenv/versions/wat/lib/python3.8/site-packages/threadpoolctl.py", line 400, in match_module_callback
    self._make_module_from_path(filepath)
  File "~/.pyenv/versions/wat/lib/python3.8/site-packages/threadpoolctl.py", line 515, in _make_module_from_path
    module = module_class(filepath, prefix, user_api, internal_api)
  File "~/.pyenv/versions/wat/lib/python3.8/site-packages/threadpoolctl.py", line 606, in __init__
    self.version = self.get_version()
  File "~/.pyenv/versions/wat/lib/python3.8/site-packages/threadpoolctl.py", line 646, in get_version
    config = get_config().split()
AttributeError: 'NoneType' object has no attribute 'split'

threadpoolctl info:
       filepath: ~/.pyenv/versions/3.8.0/envs/wat/lib/python3.8/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
         prefix: libgomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 8

       filepath: ~/.pyenv/versions/3.8.0/envs/wat/lib/python3.8/site-packages/scipy.libs/libopenblasp-r0-085ca80a.3.9.so
         prefix: libopenblas
       user_api: blas
   internal_api: openblas
        version: 0.3.9
    num_threads: 8
threading_layer: pthreads
```

user: abmyii
issue_24346: state: open
title: ENH Add dtype preservation to FeatureAgglomeration
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
In scope of #11000

#### What does this implement/fix? Explain your changes.
Implement dtype preservation to FeatureAgglomeration and its relevant tests

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: IvanLauLinTiong
issue_24337: state: open
title: ENH Add dtype preservation to LocallyLinearEmbedding
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
In scope of https://github.com/scikit-learn/scikit-learn/issues/11000

#### What does this implement/fix? Explain your changes.

Implement `dtype` preservation to `LocallyLinearEmbedding` and relevant tests for various methods and solvers.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: MisaOgura
issue_24321: state: open
title: ENH add dtype preservation to FactorAnalysis
body: #### Reference Issues/PRs

In scope of #11000 
Continuing work of @thibsej already started in [#13303](https://github.com/scikit-learn/scikit-learn/pull/13303)

#### What does this implement/fix? Explain your changes.

Added `dtype` preservation to `FactorAnalysis`

#### Any other comments?

The test passes, but did we also check the numerical stability? PR [#13303](https://github.com/scikit-learn/scikit-learn/pull/13303) added a test for this, ~~is this obsolete with our new `check_transformer_preserve_dtypes`?~~

- [x] No - we still need to check for numerical stability.

```py
sklearn/tests/test_common.py::test_estimators[FactorAnalysis()-check_transformer_preserve_dtypes] PASSED
```

user: svenstehle
labels: Needs Decision: Requires decision
issue_24313: state: open
title: BayesianRidge prediction standard deviation affected by uniform sample weights
body: ### Describe the bug

The standard deviation of predictions obtained by setting return_std=True on predict(), is clearly affected by uniform sample_weight vectors on fit(). A uniform sample_weight vector will act as a constant on the likelihood function, and I am therefore questioning the effect it has on the standard deviation of predictions. As shown below by the example, the scale (1 vs. 20) of the uniform sample_weight vector directly affects the scale of the standard deviations. The example is based on the Curve Fitting with Bayesian Ridge Regression [example](https://scikit-learn.org/stable/auto_examples/linear_model/plot_bayesian_ridge_curvefit.html#sphx-glr-auto-examples-linear-model-plot-bayesian-ridge-curvefit-py)).

### Steps/Code to Reproduce

```
import numpy as np
from sklearn.linear_model import BayesianRidge
import matplotlib.pyplot as plt

def func(x):
    return np.sin(2 * np.pi * x)

size = 25
rng = np.random.RandomState(1234)
x_train = rng.uniform(0.0, 1.0, size)
y_train = func(x_train) + rng.normal(scale=0.1, size=size)
x_test = np.linspace(0.0, 1.0, 100)

n_order = 3
X_train = np.vander(x_train, n_order + 1, increasing=True)
X_test = np.vander(x_test, n_order + 1, increasing=True)
reg = BayesianRidge(tol=1e-6, fit_intercept=False)

init = [1.0, 1e-3]
reg.set_params(alpha_init=init[0], lambda_init=init[1])

fig, axes = plt.subplots(1, 2, figsize=(8, 4))
for i, ax in enumerate(axes):
    if i == 0:
        reg.fit(X_train, y_train, sample_weight=np.ones(size))
        ymean, ystd = reg.predict(X_test, return_std=True)

        ax.plot(x_test, func(x_test), color="blue", label="sin($2\\pi x$)")
        ax.scatter(x_train, y_train, s=50, alpha=0.5, label="observation")
        ax.plot(x_test, ymean, color="red", label="predict mean")
        ax.fill_between(x_test, ymean - ystd, ymean + ystd, color="pink", alpha=0.5, label="predict std")
        ax.set_title('sample_weight=np.ones(size)')
        ax.set_ylim(-1.6, 1.6)
        ax.legend()
    elif i == 1:
        reg.fit(X_train, y_train, sample_weight=20*np.ones(size))
        ymean, ystd = reg.predict(X_test, return_std=True)

        ax.plot(x_test, func(x_test), color="blue", label="sin($2\\pi x$)")
        ax.scatter(x_train, y_train, s=50, alpha=0.5, label="observation")
        ax.plot(x_test, ymean, color="red", label="predict mean")
        ax.fill_between(x_test, ymean - ystd, ymean + ystd, color="pink", alpha=0.5, label="predict std")
        ax.set_title('sample_weight=20 * np.ones(size)')
        ax.set_ylim(-1.6, 1.6)
        ax.legend()

plt.tight_layout()
plt.show()
```



### Expected Results


![expected](https://user-images.githubusercontent.com/42994401/187893778-bd076956-dd22-4b53-90ea-faec7c27c98f.png)



### Actual Results


![actual](https://user-images.githubusercontent.com/42994401/187893819-3654f021-67db-4703-a2b2-cad2ae87071d.png)



### Versions

```shell
System:
    python: 3.8.5 (default, Sep  3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\AAA\anaconda3\python.exe
   machine: Windows-10-10.0.19041-SP0

Python dependencies:
          pip: 20.2.4
   setuptools: 50.3.1.post20201107
      sklearn: 0.24.0
        numpy: 1.22.4
        scipy: 1.5.2
       Cython: 0.29.21
       pandas: 1.1.3
   matplotlib: 3.3.2
       joblib: 0.17.0
threadpoolctl: 2.1.0

Built with OpenMP: True
```

user: gknilsen
labels: Needs Investigation: Issue requires investigation
issue_24312: state: open
title: Coordinate with other scientific python projects to use shared tools and policy to upload nightly build wheels
body: See the motivation in this discussion: https://discuss.scientific-python.org/t/interest-in-github-action-for-scipy-wheels-nightly-uploads-and-removals/397

- Draft shared Github Actions config: https://github.com/matthewfeickert/example-nightly-wheels
- New github dedicated org/repo: https://github.com/scientific-python/nightly-wheels/ 
user: ogrisel
issue_24287: state: open
title: Enhance and support the common-pitfalls page with referred published references.
body: ### Describe the issue linked to the documentation

The documentation comes with a guide called  [Common pitfalls and recommended practices](https://scikit-learn.org/stable/common_pitfalls.html). However, there are no single references on why things are right or wrong, instead presented the practitioner's best practices guide.  

### Suggest a potential alternative/fix

References to primary academic publications would strengthen the recommended practices as scikit-learn is used for serious work in industry and academics.  For example, importance of splitting strategies,

* Xu, Y., Goodacre, R. On Splitting Training and Validation Set: A Comparative Study of Cross-Validation, Bootstrap and Systematic Sampling for Estimating the Generalization Performance of Supervised Learning. J. Anal. Test. 2, 249–262 (2018). [doi](https://doi.org/10.1007/s41664-018-0068-2)


user: msuzen
issue_24281: state: open
title: Feature Request - Parallel Coordinates Plot for GridSearch result analysis
body: ### Describe the workflow you want to enable

GridSearch result are hard to analyze expecially when `param_grid` is very large.

The current documentation show usages of:
- [matrix_plot/pivot](https://scikit-learn.org/stable/_images/sphx_glr_plot_successive_halving_heatmap_001.png). That can visualize the relationship beetween 2 params for 1 metrics (2D).
- [line plot](https://scikit-learn.org/stable/_images/sphx_glr_plot_multi_metric_evaluation_001.png). Each line represent a 1D (1 params vs 1 metrics) relationship and by taking multiple line when can visualize at least 2D relationship, but this became mess for more.
- [box plot](https://scikit-learn.org/stable/_images/sphx_glr_plot_compare_reduction_001.png). Same conclusion as for line plot but using box instead.

None of these plot can represent nicely more then a 2D relationship. 3D, 4D and more can be visualize by making a bunch of basic 2D plot, but the number plot will easily became huge and we lost relationship beetween some `params`. This get even worse with `RandomizedSearchCV` as the params are not evenly compute.

One common way to represent N-D relationship without loss is to use **Parallel Coordinates Plot**:

![Example of Parallel Coordinates Plot of the proposed implementation](https://user-images.githubusercontent.com/11278197/187024865-d6183eac-8cac-45fd-a629-2a2e84daffe2.png)

The current implementation of PC Plot are:
- [Plotly express](https://plotly.com/python/parallel-coordinates-plot/). But it don't integrate well with the `param_grid` format. Plus is not matplotlib compatible.
- Tensorboard and deep learning experiement tracker (wandb ...). Same don't integrate weel, not matplotlib stack and need even more conversion work and a web server (or cloud account :()
- Handmade matplotlib implementation. Matplotlib compatible, but at the time we don't have any easy to use implementation from common package. + with `param_grid` format compatibility.

### Describe your proposed solution

I presently use my own implementation of matplotlib PC Plot (see the image above) for my work.
The current interface is:
```python
def plot_parallel_coordinates(
        grid_model: GridSearchCV, 
        params: list[str] = ["params"],
        scoring: str = ["mean_test_score"]
        cmap : Optional[str] = None,
        ax: Optional[plt.Axes] = None) -> plt.Axes
```

If think the community could benefith from using a common and scikit-learn integrated implementation (for example in `metrics.ParallelCoordinatesPlot` or `inspection.ParallelCoordinatesPlot` and with `plot_parallel_coordinates` function)

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Please give 👍 if you think this could be benefith, so we can start discuss implementation.
user: julien-blanchon
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_24247: state: open
title: Add RepeatedStratifiedGroupKFold
body: ### Describe the workflow you want to enable

Building off conversation #13621 and work already done in #18649, I'd like to add an implementation of `RepeatedStratifiedGroupKFold`.

### Describe your proposed solution

See the implementation in #24227. Then `RepeatedStratifiedGroupKFold` could be used similar to below:

```python
  >>> import numpy as np
  >>> from sklearn.model_selection import RepeatedStratifiedGroupKFold
  >>> X = np.random.randn(10, 1)
  >>> y = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])
  >>> groups = np.array([1, 1, 2, 2, 2, 3, 4, 4, 5, 5])
  >>> rsgkf = RepeatedStratifiedGroupKFold(n_splits=3, n_repeats=2, random_state=42)
  >>> for train_idxs, test_idxs in rsgkf.split(X, y, groups):
  ...     # print the group assignment for the train/test indices
  ...     print("TRAIN:", groups[train_idxs], "TEST:", groups[test_idxs])
  ...     X_train, X_test = X[train_idxs], X[test_idxs]
  ...     y_train, y_test = y[train_idxs], y[test_idxs]
  TRAIN: [2 2 2 4 4 5 5] TEST: [1 1 3]
  TRAIN: [1 1 3 4 4 5 5] TEST: [2 2 2]
  TRAIN: [1 1 2 2 2 3] TEST: [4 4 5 5]
  TRAIN: [1 1 4 4 5 5] TEST: [2 2 2 3]
  TRAIN: [2 2 2 3 4 4 5 5] TEST: [1 1]
  TRAIN: [1 1 2 2 2 3] TEST: [4 4 5 5]
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: arvkevi
issue_24243: state: open
title: TimeSeriesSplit add skip parameter
body: ### Describe the workflow you want to enable

Dear sklearn community, I want to make an hour ahead forecast on a timeseries of at least a year with an interval of 5 minutes. To do CV, I can use TimeSeriesSplit with test_size = 12. I want to do this for many different times of the day, so I could use n_splits = 24 and have a split for each hour of the day. Then I want this for multiple days, so I could do n_splits = 24 * nr_days. My problem is that I want to randomly test a few hours a day over many days of the year, to see how my forecast does on different periods of the year. To then make n_splits = 24 * 365 is too many splits.

### Describe your proposed solution

I propose a `skip` parameter, that allows skipping a number of samples before making the next split. This can be used to reduce the number of splits, while still allowing to cover a large time-period with splits.

A few examples:
Data:
| index      | foo |
|------------|-----|
| 2022-01-01 | 1   |
| 2022-01-02 | 2   |
| 2022-01-03 | 3   |
| 2022-01-04 | 4   |
| 2022-01-05 | 5   |
| 2022-01-06 | 6   |
| 2022-01-07 | 7   |
| 2022-01-08 | 8   |
| 2022-01-09 | 9   |
| 2022-01-10 | 10 |

test_size=1, max_train_size=1, n_splits=3, gap=1, skip=1

| train | test |
|-------|-----|
| 8     | 10    |
| 6     | 8      |
| 4     | 6      |

test_size=1, max_train_size=1,   n_splits=3, gap=1, skip=2
| train | test |
|-------|-----|
| 8     | 10   |
| 5     | 7    |
| 2     | 4    |

test_size=2, max_train_size=1, n_splits=3, gap=1, skip=2
not enough data

test_size=2, max_train_size=1, n_splits=2, gap=1, skip=2
| train | test  |
|-------|------|
| 7     | 9, 10 |
| 3     | 5, 6  |

test_size=2, max_train_size=1, n_splits=2, gap=1, skip=2
| train | test  |
|-------|------|
| 7     | 9, 10 |
| 3     | 5, 6  |

test_size=2, n_splits=2, gap=2, skip=2
| train               | test  |
|----------------|------|
| 1, 2                | 5, 6  |
| 1, 2, 3, 4, 5, 6 | 9, 10 |

### Describe alternatives you've considered, if relevant

I could generate all those splits, then implement a wrapper in my own codebase that does the skipping.

### Additional context

I already have this solution implemented locally. The important changes to the function `TimeSeriesSplit.split()` are:

```
        if n_samples - gap - ((test_size + skip) * n_splits) < 0:
            raise ValueError(
                f"Too many splits={n_splits} for number of samples"
                f"={n_samples} with test_size={test_size}, gap={gap} and skip={skip}."
            )
```

```
test_starts = range(n_samples - n_splits * (test_size + skip) + skip, n_samples, test_size + skip)
```

I'd be happy to make a PR if you think this can be included. Thank you for your time and consideration!
user: matthiasschuurmans
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_24228: state: open
title: Balltree's node_data radii seems to be incorrect
body: ### Describe the bug

The node data from constructing a BallTree seems to be incorrect. Each parent node's radius should always be greater than or equal to both its child's radiuses, since the radius is that of the smallest hypersphere containing all the points within that node, and the children of a node is just a subset of that node. Therefore if a child's radius is higher than its parents, it could just adopt the hypersphere of its parent as its current sphere to have a smaller radius.

That being said, I did some testing internally and it is very common for either or both children of a node in a BallTree to have a higher radius than their parent! Can someone point out what I might be doing wrong here, or is the BallTree radius simply an approximation? I don't think it is either, since we see [here](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neighbors/_ball_tree.pyx#L92) that the radius of a node is computed as the largest distance from its centroid (mean of all its vectors) to any vector within the node. 

Edit: I think I understand why child nodes could potentially have a larger radius than their parent, due to the fact that we use a new centroid based on the points within a node each time (instead of comparing to the parent centroid). It's possible that we end up using a worse centroid which nets a larger radius. However, I still think the radius calculation is off - see my follow up comment below.

### Steps/Code to Reproduce
```
from sklearn.neighbors import BallTree    
import numpy as np

np.random.seed(3)
Z = np.random.random((1000, 512)) 
model = BallTree(Z, leaf_size=1, metric='l2')
(
    _,
    idx_array,
    node_data,
    _,
) = model.get_arrays()

root = node_data[0]
lchild = node_data[1]
rchild = node_data[2]

_, _, _, root_radius = root
_, _, _, left_radius = lchild
_, _, _, right_radius = rchild


if left_radius > root_radius:
    print(left_radius, root_radius)
if right_radius > root_radius:
    print(right_radius, root_radius)
```

### Expected Results

Nothing should be printed.

### Actual Results

`6.93039685892859 6.923245275411173`


### Versions

```shell
System:
    python: 3.8.12 (default, Oct 12 2021, 06:23:56)  [Clang 10.0.0 ]
executable: /opt/miniconda3/envs/models/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 21.2.4
   setuptools: 62.1.0
      sklearn: 0.24.2
        numpy: 1.20.3
        scipy: 1.7.1
       Cython: 0.29.24
       pandas: 1.1.5
   matplotlib: 3.4.2
       joblib: 1.0.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

user: jihan-yin
labels: Needs Investigation: Issue requires investigation
issue_24227: state: open
title: [MRG] Add RepeatedStratifiedGroupKFold
body: #### Reference Issues/PRs
This functionality was discussed in #13621.

#### What does this implement/fix? Explain your changes.
This adds a splitter class to `model_selection` that repeats `StratifiedGroupKFold` n times.

#### Any other comments?


user: arvkevi
issue_24216: state: open
title: [WIP] example: showcase advantage of balanced_accuracy_score for imbalanced data
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.

Adds a documentation example showing a comparison between the metrics `accuracy_score` and `balanced_accuracy_score`.
By means of simulating different ground truth labels distribution, the example is supposed to showcase the advantage of `balanced_accuracy_score` when working with imbalanced data. 

#### Any other comments?
This is a first WIP stub showing the basic idea of the intended changes.
If reviewers agree this could be an interesting addition, I'd be happy to expand with more details and polish the rough edges.
I'd appreciate the feedback :)

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: fabridamicelli
issue_24199: state: open
title: Allow X in func or transformer in TransformedTargetRegressor
body: ### Describe the workflow you want to enable

I would like to enable target transformation based on X in data. 

This will come in handy when predicting a quantity that has a natural dependency on a feature. E.g. target value is summed salary of a household and the function is to divide by number of adults in household. This creates a natural scaling of the target but requires features in training data to accomplish. 

In this case the proposed solution would look something like this

```
def func(X, y):
   return y / x[1]

def inverse(X, y):
   return y * X[1]
```

### Describe your proposed solution

Check signature on func and transformer calls to see if it requires X as parameter and if so add it to function calls similar to https://github.com/scikit-learn/scikit-learn/pull/21935

Here is a hacky solution I made myself to get it to work

```
class TargetValueTransformer(TransformedTargetRegressor):
    def __init__(
        self,
        regressor=None,
        *,
        transformer=None,
        func=None,
        inverse_func=None,
        check_inverse=False,
    ):        
        super(TargetValueTransformer, self).__init__(regressor,        
        transformer=transformer,
        func=func,
        inverse_func=inverse_func,
        check_inverse=check_inverse,
)
      
    def _fit_transformer(self, y):
        pass
        
    def fit(self, X, y):
        self.regressor.fit(X, self.func(X, y))
        return self
    
    def predict(self, X):
        return self.inverse_func(X, self.regressor.predict(X))
```
### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: mattiasAngqvist
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_24188: state: open
title: FEA Add sample variance support for `GaussianProcessRegressor.fit`
body: add sample variance option in "fit" interface, make it more coherent/consistent with other estimators (i.e. "sample_weight" option in other estimators)

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

See also #7975, #7985, #7821

#### What does this implement/fix? Explain your changes.
I add the sample_variance option in fit interface

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: wen-zheng
issue_24187: state: open
title: Incremental versions of Existing Clustering Indices (metrics)
body: ### Describe the workflow you want to enable

There currently exists 3 CVIs (cluster validity indices) in the main branch. These metrics are very useful for indicating performance of unsupervised clustering methods and add value to the module. However, these methods suffer from high computational complexity which can make them intractable for large datasets and online implementations. Silhouette, Davies-Bouldin, and Calinski-Harabasz have O(n^2), O(nlogn), and O(nlogn) complexity, respectively. 

Within the past few years, incremental versions of these algorithms have been introduced. These iCVIs replicate identical results of the batch (original) versions but in an online and more efficient way. The incremental versions of those existing CVIs have time complexities of O(n) for each. This has obvious benefit for large datasets and online implementation. 

While the incremental versions of these indices are themselves new and not highly cited, they are simply improvements to methods which already exist in this repository and would extend viability to a larger range of applications. 

### Describe your proposed solution

Implement incremental versions of the Silhouette, Davies-Bouldin, and Calinski-Harabasz methods within the metrics submodule. Each method would necessarily be a class with an incremental update method.

Given the go-ahead, I will submit a pull request with these methods.

### Describe alternatives you've considered, if relevant

Simply continue using the batch versions and deal with the time complexity

### Additional context

https://arxiv.org/pdf/1801.02937.pdf
https://ieeexplore.ieee.org/document/8970493
user: NiklasMelton
issue_24172: state: open
title: SGDClassifier initial lr undocumented
body: ### Describe the issue linked to the documentation

The [documentation for SGDClassifier](https://scikit-learn.org/stable/modules/sgd.html#sgd) says that with `learning_rate='optimal'` the initial learning rate `t0` is chosen with a heuristic proposed by Léon Bottou that "can be found in `_init_t` in BaseSGD".

However currently this method [does not exist in BaseSGD](https://github.com/scikit-learn/scikit-learn/search?q=_init_t).

### Suggest a potential alternative/fix

The calculation of t0 should be documented either by adding the formula, pointing to the correct method in the code, and/or specifying exactly what paper of Bottou is being referred to.
user: morrisalp
issue_24154: state: open
title: joblib and threadpoolctl dependencies are missing in setup.py and pyproject.toml
body: ### Describe the bug

README.rst lists them as dependencies, but they aren't in setup.py and pyproject.toml

### Steps/Code to Reproduce

n/a

### Expected Results

n/a

### Actual Results

n/a

### Versions

```shell
n/a
```

user: yurivict
issue_24138: state: open
title: sklearn.svm.SVR use more RAM memory on newer versions
body: ### Describe the bug

Hi, I've previously reported a bug that [RandomForestClassifier](https://github.com/scikit-learn/scikit-learn/issues/24124) consumes more memory on newer versions, and I've also noticed that the svm API also consumes more memory on newer versions. The detailed test information is as follows.
|Memory| Version|
|--|--|
|274MB|1.0.1|
|276MB|0.20.3|
|180MB|0.19.2|


### Steps/Code to Reproduce

[Download dataset](https://drive.google.com/drive/folders/1tI0h0ZoL8bnvlZyVsa0-QPdKTcrZ_kVv?usp=sharing)

```python
import math
import pandas as pd
import numpy as np
import random
import torch
from sklearn.svm import SVR
from sklearn.ensemble import BaggingRegressor

def random_init(**kwargs):
    random.seed(kwargs['seed'])
    np.random.seed(kwargs['seed'])
    torch.manual_seed(kwargs['seed'])
    torch.cuda.manual_seed(kwargs['seed'])
    torch.backends.cudnn.deterministic = True
def load_data(df,cv=False,target=False,**kwargs):
    num_samples = len(df)
    sample_size = sum([len(args['categories'][c]) for c in args['categories']]) + len(args['num_feats'])
    dataset = torch.zeros((num_samples,sample_size),dtype=torch.float)
    idx = 0
    for c in args['cat_feats']:
        for i in range(len(args['categories'][c])):
            dataset[np.array(df[c])==args['categories'][c][i],idx] = 1.0
            idx += 1
    for n in args['num_feats']:
        dataset[:,idx] = torch.from_numpy(np.array(df[n]))
        idx += 1
    if target:
        targets = torch.from_numpy(np.array(df['target']))
    else:
        targets = None
    
    if cv == False:
        return dataset, targets

    idx = [i for i in range(num_samples)]
    random.shuffle(idx)
    trainset = dataset[idx[0:int(num_samples*(1-kwargs['cv_percentage']))]]
    traintargets = targets[idx[0:int(num_samples*(1-kwargs['cv_percentage']))]]
    validset = dataset[idx[int(num_samples*(1-kwargs['cv_percentage'])):]]
    validtargets = targets[idx[int(num_samples*(1-kwargs['cv_percentage'])):]]
    return trainset, validset, traintargets, validtargets  
def get_stats(trainset):
    mean = torch.mean(trainset,dim=0)
    std = torch.std(trainset,dim=0)
    for i in range(trainset.shape[1]):
        if ((trainset[:,i]==0) | (trainset[:,i]==1)).all():
            mean[i] = 0.5
            std[i] = 1.0
    return mean, std
#Global arguments
args = {
    'cv_percentage': 0.1,
    'seed': 0,
    }
# Load data
random_init(**args)
train_data = pd.read_csv('train.csv')
test_data = pd.read_csv('test.csv')
args['cat_feats'] = [c for c in np.sort(train_data.columns) if 'cat' in c]
args['num_feats'] = [c for c in np.sort(train_data.columns) if 'cont' in c]
args['categories'] = {c: np.unique(train_data[c]) for c in args['cat_feats']}
testset, _ = load_data(test_data,cv=False,target=False,**args)
trainset, validset, traintargets, validtargets = load_data(train_data,cv=True,target=True,**args)
args['mean'],args['std'] = get_stats(trainset)
val_pred = {}
test_pred = {}
# Bagged SVRs
import tracemalloc
tracemalloc.start()
print('Bagged SVRs...')
# 0.19.2 180MB
# 0.20.3 276
# SVR arguments
args = {**args,**{
    'kernel': 'linear',
    'n_estimators': 3,
    'max_samples': 20000,
    'max_features':50
    }}
random_init(**args)
svrs = BaggingRegressor(SVR(kernel=args['kernel'],C=1.0, epsilon=0.2),
                        n_estimators=args['n_estimators'],max_samples=args['max_samples'],max_features=args['max_features'])
svrs.fit(((trainset-args['mean'])/args['std']).numpy(),traintargets.numpy())
val_pred['svrs'] = svrs.predict(((validset-args['mean'])/args['std']).numpy())
test_pred['svrs'] = svrs.predict(((testset-args['mean'])/args['std']).numpy())
currentSVR, peakSVR = tracemalloc.get_traced_memory()
print("SVR current memory usage is {",currentSVR /1024/1024,"}MB; SVR Peak memory was :{",peakSVR / 1024/1024,"}MB")
```

### Expected Results

same memory usage

### Actual Results

new version use more memory.

### Versions

```shell
1.0.1, 0.20.3, 0.19.2
```

user: Piecer-plc
labels: Needs Investigation: Issue requires investigation
issue_24121: state: open
title: Adds gain scoring metrics
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->




#### What does this implement/fix? Explain your changes.

This PR adds Precision Gain, Recall Gain and Fscore Gain metrics to scikit-learn.

The metrics are described in this [paper](https://papers.nips.cc/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf) by Peter Flach and Meelis Kull at the University of Bristol, UK.

In short, these improved metrics have desirable properties such as meaningful linear interpolation of PR curves and a universal baseline across tasks. 

I've added test coverage for the gain metrics and utilized existing scikit-learn Precision/Recall functionality where-ever possible.


The new methods, which can be imported `from sklearn.metrics` are:
```
- f1_gain_score
- fbeta_gain_score
- precision_recall_fgain_score_support
- precision_gain_score
- recall_gain_score
```


#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Request for this feature was referenced in https://github.com/scikit-learn/scikit-learn/issues/21391#issuecomment-962592956 by @glemaitre 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: bradleyfowler123
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_24119: state: open
title: FIX Correct `GaussianMixture.weights_` normalization
body: #### Reference Issues/PRs
Fixes #24085 


#### What does this implement/fix? Explain your changes.
Weights that were normalized using `n_samples` now use `weights_.sum()`

#### Any other comments?

user: kasmith11
labels: Moderate: Anything that requires some knowledge of conventions and best practices
Needs Decision: Requires decision
issue_24107: state: open
title: Gaussian Process regression get_params set_params does not yield same predictor? 
body: ### Describe the bug

This is on version 1.1.1

I *think* the example below is the correct way of setting GPR params saved from a previous .fit() tuning via the optimizer. Reading through the code I don't yet understand why the parameters `self.L_` and `self.alpha_` are not being updated but maybe am missing something in my reading.

See below for the example. Everything passes except the last line. 

### Steps/Code to Reproduce

```python
import sys

import numpy as np
from scipy.linalg import cholesky
from sklearn.gaussian_process import GaussianProcessRegressor, kernels
from sklearn.gaussian_process.kernels import *

np.random.seed(0)

X = np.random.randn(100, 3)
y = np.random.randn(100, 1)

KERNEL = Matern()
# make it once, fit and get the params
gp_ = GaussianProcessRegressor(kernel=KERNEL, normalize_y=True, alpha=1e-5)
gp_.fit(X, y)
params = gp_.kernel_.get_params(deep=False)  # tried deep=True but doesn't matter

# make it again, don't fit, but set the params
gp = GaussianProcessRegressor(kernel=KERNEL, normalize_y=True, alpha=1e-5)
gp.optimizer = None  # this should make fit do very little
gp.fit(X, y)  # this creates kernele

# do it again for good measure? Maybe we need to do this several times to trigger it?
gp.kernel_.set_params(**params)
gp.fit(X, y)  # do this again to now recreate L_
gp.kernel_.set_params(**params)


def get_K_L_(self):
    K = self.kernel_(self.X_train_)
    K[np.diag_indices_from(K)] += self.alpha
    L_ = cholesky(K, lower=True)
    return K, L_


Ka, La = get_K_L_(gp_)
Kb, Lb = get_K_L_(gp)

a = gp_.__getstate__()
b = gp.__getstate__()

# here you will see alpha_ and L_ are not the same. Something is not updating them correctly?
# see https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/gaussian_process/_gpr.py#L322
for k in a:
    if isinstance(a[k], np.ndarray):
        print(k, type(a[k]), np.all(a[k] == b[k]))


assert np.all(gp_.X_train_ == gp.X_train_)  # passes
assert gp_.alpha == gp.alpha  # passes
assert np.allclose(Ka, Kb), 'Ka Kb do not match'  # passes
assert np.allclose(La, Lb), 'La Lb do not match'  # passes
# it is bizarre that the output of the kernels matchs but not L_????
assert np.allclose(gp.L_, gp_.L_), 'L_ do not match'  # FAILS
```

### Expected Results

No exceptions. The gp and gp_ should return the same results, have same state (except for the optimizer).

### Actual Results

```
_y_train_mean <class 'numpy.ndarray'> True
_y_train_std <class 'numpy.ndarray'> True
X_train_ <class 'numpy.ndarray'> True
y_train_ <class 'numpy.ndarray'> True
L_ <class 'numpy.ndarray'> False
alpha_ <class 'numpy.ndarray'> False
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-5-71be2b0a00fc> in <module>
     53 assert np.allclose(La, Lb), 'La Lb do not match'  # passes
     54 # it is bizarre that the output of the kernels matchs but not L_????
---> 55 assert np.allclose(gp.L_, gp_.L_), 'L_ do not match'  # FAILS

AssertionError: L_ do not match
```

### Versions

```shell
System:
    python: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0]
executable: /home/cottrell/anaconda3/envs/3.10.4/bin/python
   machine: Linux-5.15.0-43-generic-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.1.1
          pip: 22.1.2
   setuptools: 61.2.0
        numpy: 1.23.1
        scipy: 1.8.1
       Cython: 0.29.30
       pandas: 1.4.3
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/cottrell/anaconda3/envs/3.10.4/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-742d56dc.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: Haswell
    num_threads: 12

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/cottrell/anaconda3/envs/3.10.4/lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 12

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/cottrell/anaconda3/envs/3.10.4/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-8b9e111f.3.17.so
        version: 0.3.17
threading_layer: pthreads
   architecture: Haswell
    num_threads: 12

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/cottrell/anaconda3/envs/3.10.4/lib/python3.10/site-packages/opencv_python_headless.libs/libopenblas-r0-f650aae0.3.3.so
        version: None
threading_layer: disabled
   architecture: Haswell
    num_threads: 1
```

user: cottrell
labels: Needs Investigation: Issue requires investigation
issue_24106: state: open
title: FEA Add `ArccosDistance` (continued)
body: #### Reference Issues/PRs

This is the continuation of #7829. I worked on this about a year ago and remember that it worked and I was (nearly) ready to submit a PR, but I was too busy at work to actually do it. Now, @ArturoAmorQ seems to plan to redo all of my work in #24099, which prompted me to just submit what I have.

#### What does this implement/fix? Explain your changes.

As originally described by @lmcinnes in #7829:

> Many algorithms, such as word2vec result in nearest neighbor computations based on cosine similarity. Unfortunately, since cosine (dis)similarity is not a metric it can't be used with kd-trees and ball-trees. This means that algorithms that make use of these structures (e.g. DBSCAN clustering, fast t-SNE, etc.) can't operate with regard to the "appropriate" (dis)similarity measure. Here we add angular (or arccos) distance which is the natural metric analogue of cosine dissimilarity to the valid metrics used for kd-trees and ball-trees. Credit for this work belongs to @brunoalano who submitted a similar change to hdbscan.

#### Any other comments?

Due to the urgency, I have not checked whether the branch ticks all the marks; I will correct as needed.

@ArturoAmorQ, please have a look at this branch before investing more effort in #24099.
user: jgonggrijp
issue_24086: state: open
title: Add other connectivity definitions to grid_to_graph function
body: ### Describe the workflow you want to enable

The current grid_to_graph function only defines voxel neighbors with the 6-connectivity definition. I would like to add 18 and 26 connectivity. (https://en.wikipedia.org/wiki/Pixel_connectivity#26-connected)

### Describe your proposed solution

I already made a [PR](https://github.com/scikit-learn/scikit-learn/pull/16405#event-7047878910) that addresses this change, but it was closed and i was encouraged to make an issue first.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: tathey1
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_24085: state: open
title: Weights are being normalized using number of samples as opposed to sum in GaussianMixture
body: ### Describe the bug

Weights are being normalized at Line https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/mixture/_gaussian_mixture.py#L718 using `n_samples`. It should be done using `weights.sum()` as
done in `_m_step()` here: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/mixture/_gaussian_mixture.py#L756.

### Steps/Code to Reproduce

Weights are being normalized at Line https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/mixture/_gaussian_mixture.py#L718 using `n_samples`. It should be done using `weights.sum()` as
done in `_m_step()` here: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/mixture/_gaussian_mixture.py#L756.

### Expected Results

Correct weights.

### Actual Results

Incorrect weights.

### Versions

```shell
System:
    python: 3.9.13 (main, May 24 2022, 21:28:31)  [Clang 13.1.6 (clang-1316.0.21.2)]
executable: /Users/kshitijgoel/Documents/main/code.nosync/self_organizing_gmm/.venv/bin/python
   machine: macOS-12.4-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.1.1
          pip: 22.2.1
   setuptools: 62.3.2
        numpy: 1.23.1
        scipy: 1.8.1
       Cython: 0.29.30
       pandas: 1.4.3
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: False

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /Users/kshitijgoel/Documents/main/code.nosync/self_organizing_gmm/.venv/lib/python3.9/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.20
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /Users/kshitijgoel/Documents/main/code.nosync/self_organizing_gmm/.venv/lib/python3.9/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.17
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8
```

user: kshitijgoel007
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_24079: state: open
title: Add `sample_weight` support to PCA
body: #### What does this implement/fix? Explain your changes.
This PR adds support for the `sample_weight` parameter in the `PCA` class, AKA weighted PCA. This allows for e.g. performing PCA on distinct data with aggregated counts. 


#### Any other comments?

I added 2 new unit tests to help validate the functionality, in addition to these tests I ran a battery of my own randomized instances to help ensure things are working properly.

A few things that probably need some eyes:

1) The `fit_transform` logic is a bit tricky with sample weights. I added comments to help clarify. Let me know if they are too verbose.
2) If an already normalized `sample_weight` vector is provided, it's ambiguous how many samples there truly are. I have somewhat of a heuristic defined in `_sample_count` to determine the true sample count, and in this situation it would default to `n_samples`.

Feedback, questions, or comments are welcome.
user: nogilnick
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_24053: state: open
title: EHN: RadiusNeighborRegressor speedup
body: #### What does this implement/fix? Explain your changes.

When working with low-dimensional data the neighborhood queries are very fast and most of the time is spent iterating through the neighborhood to compute the predictions since the neighborhoods might not have the same length it employs a different mechanism than the `KNNRegressor`.

This PR converts the loop operations to sparse matrix multiplications, providing 10x speed on low-dimensional datasets without performance decrease on higher dimensions (where most of the time is spent on neighbor's query).

Relevant results from ASV:

```
THIS BRANCH:

[16.67%] ··· neighbors.RadiusNeighborsRegressorBenchmark.peakmem_predict                                                                             ok
[16.67%] ··· =========== ============== ========= ==========
             --                   dimension / n_jobs        
             ----------- -----------------------------------
              algorithm   very-low / 4   low / 4   high / 4 
             =========== ============== ========= ==========
                brute         788M         777M      686M   
               kd_tree       87.9M        91.7M      128M   
              ball_tree      87.4M        91.4M      127M   
             =========== ============== ========= ==========

[33.33%] ··· neighbors.RadiusNeighborsRegressorBenchmark.time_predict                                                                                ok
[33.33%] ··· =========== ============== ========== ============
             --                    dimension / n_jobs          
             ----------- --------------------------------------
              algorithm   very-low / 4   low / 4     high / 4  
             =========== ============== ========== ============
                brute       825±30ms     847±30ms    920±30ms  
               kd_tree      15.1±3ms     36.3±6ms    211±50ms  
              ball_tree     63.7±8ms     785±60ms   4.31±0.03s 
             =========== ============== ========== ============

MAIN BRANCH:

[66.67%] ··· neighbors.RadiusNeighborsRegressorBenchmark.peakmem_predict                                                                             ok
[66.67%] ··· =========== ============== ========= ==========
             --                   dimension / n_jobs        
             ----------- -----------------------------------
              algorithm   very-low / 4   low / 4   high / 4 
             =========== ============== ========= ==========
                brute         817M         729M      641M   
               kd_tree       88.1M         92M       128M   
              ball_tree      87.9M        91.7M      127M   
             =========== ============== ========= ==========
             
[83.33%] ··· neighbors.RadiusNeighborsRegressorBenchmark.time_predict                                                                                ok
[83.33%] ··· =========== ============== ========== ============
             --                    dimension / n_jobs          
             ----------- --------------------------------------
              algorithm   very-low / 4   low / 4     high / 4  
             =========== ============== ========== ============
                brute       906±40ms     939±30ms   1.02±0.04s 
               kd_tree      116±20ms     130±20ms    320±60ms  
              ball_tree     166±20ms     890±40ms   4.31±0.04s 
             =========== ============== ========== ============
```

#### Any other comments?

After these changes, profiling indicates that more than half of the time (for low-dimension datasets) is spent on the `_get_weights` function. I was able to get a speedup on it, but the behavior changed a little bit when there's a point identical to the training set. Hence, I'm not pushing this change, but additional suggestions are welcome.

user: JoOkuma
issue_24034: state: open
title: Add a page on the website to provide directions to users on how to reach out
body: Add a page on the website to provide directions to users on how to reach out.

This solve the request #23840

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
It is adding a new page (with several links provided on #23840 and a link to it on the menu).

#### Any other comments?
This is my first time solving an issue so I'm not really sure if I'm doing it correctly :)

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: rriccilopes
issue_24024: state: open
title: Optimize inference time for sklearn models in real time for single example
body: ### Describe the bug

It is a multi-class classification model with sklearn. 

I am using `OneVsOneClassifier` model to train and predict `150 intents`. Its a multi-class classification problem.

**Data:**

    text          intents
    
    text1         int1
    text2         int2

I convert these intents in labels using:

    le = LabelEncoder()
    y_train = le.fit_transform(y_train)
    y_test = le.fit_transform(y_test)

 **Expectation:**

Without changing the training pipeline or parameters, note the inference time. Currently, it's slow, ~1second for 1 inference. 
`To Check if` : this is expected; there is a problem in setting (e.g., the kernels, the libarries); there is a bug in inference code

**Code:**

```python
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.compose import ColumnTransformer
    from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
    from sklearn.pipeline import Pipeline
    from sklearn.svm import SVC,LinearSVC
    
    def create_pipe(clf):
        
        # Each pipeline uses the same column transformer.  
        column_trans = ColumnTransformer(
                [('Text', TfidfVectorizer(), 'text')
                 ],
                remainder='drop') 
        
        pipeline = Pipeline([('prep',column_trans),                     
                             ('clf', clf)])
         
        return pipeline
    
    def fit_and_print(pipeline):
        
        pipeline.fit(X_train, y_train)
        y_pred = pipeline.predict(X_test)
    
        print(metrics.classification_report(y_test, y_pred, 
                                            target_names=le.classes_, 
                                            digits=3))
    clf = OneVsOneClassifier(LinearSVC(random_state=42, class_weight='balanced'))
    pipeline = create_pipe(clf)
    %time fit_and_print(pipeline)
    
    # convert input to df
    
    def create_test_data(x):
        d = {'text' : x}
        df = pd.DataFrame(d, index=[0])
        return df
    
    revs=[]
    for idx in [948, 5717, 458]:
         cur = test.loc[idx, 'text']
         revs.append(cur)
    print(revs) 
    
    revs=sam['text'].values
    
    %%time
    for rev in revs:
        c_res = pipeline.predict(create_test_data(rev))
        print(rev, '=', labels[c_res[0]])
```

For inferencing on `one data at at time` its taking `~1 sec.` As in real, live data one data point will be given to inferencing model.

But when I do `batch inferencing` it is optimised, as for `60 samples` it just takes `~1 second.`

**Is there a way to improve inference time on 1 sample? Or is this the expected behaviour?**


### Steps/Code to Reproduce

     from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.compose import ColumnTransformer
    from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
    from sklearn.pipeline import Pipeline
    from sklearn.svm import SVC,LinearSVC
    
    def create_pipe(clf):
        
        # Each pipeline uses the same column transformer.  
        column_trans = ColumnTransformer(
                [('Text', TfidfVectorizer(), 'text')
                 ],
                remainder='drop') 
        
        pipeline = Pipeline([('prep',column_trans),                     
                             ('clf', clf)])
         
        return pipeline
    
    def fit_and_print(pipeline):
        
        pipeline.fit(X_train, y_train)
        y_pred = pipeline.predict(X_test)
    
        print(metrics.classification_report(y_test, y_pred, 
                                            target_names=le.classes_, 
                                            digits=3))
    clf = OneVsOneClassifier(LinearSVC(random_state=42, class_weight='balanced'))
    pipeline = create_pipe(clf)
    %time fit_and_print(pipeline)
    
    # convert input to df
    
    def create_test_data(x):
        d = {'text' : x}
        df = pd.DataFrame(d, index=[0])
        return df
    
    revs=[]
    for idx in [948, 5717, 458]:
         cur = test.loc[idx, 'text']
         revs.append(cur)
    print(revs) 
    
    revs=sam['text'].values
    
    %%time
    for rev in revs:
        c_res = pipeline.predict(create_test_data(rev))
        print(rev, '=', labels[c_res[0]])

### Expected Results

Time for single example should be in milliseconds.

### Actual Results

Time for single example is 1.33 seconds

### Versions

```shell
System:
    python: 3.6.13 |Anaconda, Inc.| (default, Jun  4 2021, 14:25:59)  [GCC 7.5.0]
executable: /opt/conda/bin/python
   machine: Linux-3.10.0-1160.25.1.el7.x86_64-x86_64-with-debian-stretch-sid

Python dependencies:
          pip: 21.2.3
   setuptools: 57.4.0
      sklearn: 0.24.2
        numpy: 1.19.5
        scipy: 1.5.4
       Cython: 0.29.24
       pandas: 1.1.5
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 3.0.0

Built with OpenMP: True
```

user: pratikchhapolika
issue_24000: state: open
title: [RFC] Modularize the tree class in both Python and Cython to enable easy extensions
body: ### Describe the workflow you want to enable

As we are waiting for a reviewer to review #22754 , @thomasjpfan suggested we just move forward with our goals of creating a package of more exotic tree splits. E.g. https://arxiv.org/abs/1909.11799. While we wait for reviewers, the suggestion is to make a package within `scikit-learn-contrib`. 

Although we would like for #22754 to be eventually merged into scikit-learn, we understand reviewer backlog is an issue. To move forward while reviews occur, we would need to subclass existing scikit-learn code. Ideally, we would like to introduce minor refactoring changes that would make this task significantly easier.

We would like to subclass directly from scikit-learn without requiring us to keep an up-to-date fork of scikit-learn with all the bug fixes and maintenance that the dev team here does. We can limit this if we modularize the Python/Cython functions inside the `sklearn/tree` module.

### Describe your proposed solution

I am proposing two refactoring modifications that have no impact on the performance of the current tree estimators in scikit-learn.

1. Refactor the `BaseDecisionTree` Python class to have the following functions that can be overridden in a subclass:

- `_set_tree_class`: sets the Tree Cython class that the Python API uses
- `_set_splitter`: sets the Splitter Cython class that the Python API uses

For example, this makes the subclassing of BaseDecisionTree cleaner: https://github.com/scikit-learn/scikit-learn/blob/de06afa78458d773d7c0482ff17a55a235d0886b/sklearn/tree/_classes.py#L410-L416.

2. Refactor the `Tree` Cython class to have the following functions:

- `_set_node_values`: transfers split node values to the storage node
- `_compute_feature_value`: uses the storage node and the input data to compute the feature value to split on

For example, see https://github.com/scikit-learn/scikit-learn/blob/de06afa78458d773d7c0482ff17a55a235d0886b/sklearn/tree/_tree.pyx#L770-L787.

3. Refactor the `TreeBuilder` Cython class to pass around a `Split` pointer, rather than the struct itself

This will enable someone to use C-level functions to pass around another struct with a similar structure as `Split`.

For example, see https://github.com/scikit-learn/scikit-learn/blob/de06afa78458d773d7c0482ff17a55a235d0886b/sklearn/tree/_tree.pyx#L193.

### Describe alternatives you've considered, if relevant

Alternatives would require maintaining a copy of the `sklearn/tree` module and keep it up-to-date w/ sklearn changes. If this was just one Cython file, I would say it is possible, but the necessary ingredients span some of the underlying private API, making this a very time-consuming task. Introducing modularity into the private API that does not impact existing performance, therefore seems to be the best path forward?

Moreover, by introducing these refactoring changes, #22754 has a smaller diff and lower-cost to review.

### Additional context

**#22754 demonstrates that there is no performance regression, or issues w/ existing DecisionTree, or RandomForest when introducing these changes.**
user: adam2392
labels: Needs Decision - Include Feature: Requires decision regarding including feature
Refactor: Code refactor
issue_23983: state: open
title: Add option to RocCurveDisplay to display the average of different length ROC curves 
body: ### Describe the workflow you want to enable

When using k-fold cross-validation the resulting ROC curves can vary in length if there are a different number of positive and/or negative samples in each fold. I would like to add an option to sklearn.metrics.RocCurveDisplay to display the average of different length ROC curves.

Section 8 of [An introduction to ROC analysis](https://ccrma.stanford.edu/workshops/mir2009/references/ROCintro.pdf) by Fawcett describes the method.

### Describe your proposed solution

```python
def average_roc_curves(roc_curves_list,):
    """
    This function takes the average of different length ROC curves returned from 
    sklearn.metrics.roc_curve. The function expects a list of fpr, tpr, and 
    thresholds. The function subsamples from these lists and returns an 
    average_fpr, average_tpr, and average_thresholds.

    Parameters
    ----------
    roc_curves_list : list
        A list of fpr, tpr, and thresholds
    Returns
    -------
    average_fpr : list
        A list of averaged false positive rates
    average_tpr : list
        A list of averaged true positive rates
    average_thresholds : list
        A list of averaged thresholds
    """
    kfolds = len(roc_curves_list)
    min_length = np.min(list(len(roc_curves_list[k][2]) for k in range(kfolds)))
    shortened_roc_curves_list = list()
    for k in range(kfolds):
        fpr, tpr, thresholds = roc_curves_list[k]
        indicies = list(i for i in range(len(thresholds)))
        selected_indicies = np.sort(np.random.choice(indicies, min_length, replace=False))
        shortened_roc_curves_list.append([fpr[selected_indicies], tpr[selected_indicies], thresholds[selected_indicies]])
    average_fpr, average_tpr, average_thresholds = np.mean(shortened_roc_curves_list, axis=0)
    return average_fpr, average_tpr, average_thresholds
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: ethanharvey98
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23970: state: open
title: Test Failed on CICD Pipeline with varied LinearRegression predictions
body: - Ref: https://stackoverflow.com/questions/73067554/test-failed-on-cicd-pipeline-with-varied-linearregression-predictions

We have a very simple use case, when we take some data `X`, `y` and fir it to `LinearRegression()` like below. and try to predict some more value.

We have set the seed to some non-zero number again mentioned below.

```py
  np.random.seed(42)
  regr = LinearRegression()
  regr.fit(x, y)
  
  pred = regr.predict(pred_set)[0]
```

The above prediction where as always same in one machine, changes when ran from different machine, specifically when we ran it in some CICD test pipeline on cloud environment.

As per my understanding, setting seed to a specific number should confirm the reproducibility, but is there anything I am missing here. Or If I need to set seed in some other way.

Kindly, let me know.

user: jyotirmay123
labels: Needs Investigation: Issue requires investigation
issue_23962: state: open
title: [WIP] Make random_state accept np.random.Generator
body: #### Reference Issues/PRs

Fixes #16988

#### What does this implement/fix? Explain your changes.

The `random_state` argument accepts [numpy.random.Generator](https://numpy.org/doc/stable/reference/random/generator.html).

#### Any other comments?

##### Context

**Update**: Please see [this comment](https://github.com/scikit-learn/scikit-learn/pull/23962#issuecomment-1195307857).

This is WIP and I discussed with @thomasjpfan that it would make sense to share the current progress to evaluate if the scope is sufficiently small for a single PR or if we need to split it.

##### Done

- Added tests for estimators
- ~Made tests for estimators pass~ (reverted)

##### Missing

- [ ] tests for splitters
- [ ] tests for other components, e.g. for creating random datasets (this will be difficult because those components need to be called, which is not possible to do in a generic way, unlike for estimators)
- [ ] documentation
- [ ] docstrings
- [ ] [SeedSequence](https://numpy.org/doc/stable/reference/random/bit_generators/generated/numpy.random.SeedSequence.html#numpy.random.SeedSequence) use for `n_jobs>1` is probably out of scope

##### Implementation

One difficulty is that `Generator` has a slightly different API than the existing `RandomState` class, namely that creating integers now happens through the `integers` method, not `randint`. We (Thomas and I) discussed 3 different approaches to support `Generators`:

1. Use an adapter with the API of `RandomState`

If `check_random_state` sees a `Generator`, it returns an adapter that supports the `randint` method with the old signature. This would be backwards compatible with all existing code but locks sklearn into the "old way". Also, the appearance of this new class could be surprising to users.

2. Use an adapter with the API of `Generator`

If `check_random_state` sees a `RandomState`, it returns an adapter that supports the `integers` method with the old signature. This would be forwards compatible with the "new way". However, it requires changing all existing calls to `randint`  and the appearance of this new class could be surprising to users.

3. Using a utility function that knows how to deal with both objects

This is the way that [scipy approached the problem](https://github.com/scipy/scipy/blob/f3732bc64fb6ffa013a75c566447c02a5448bc5e/scipy/_lib/_util.py#L449). It also requires to change all the calls to `randint` but it's more transparent than solution 2. One disadvantage is that all other sampling functions are method calls on the object, only integers require this function, which can be surprising.

In the end, we decided to go with option 3. because we assume that it worked well for scipy and should thus also serve sklearn well.

Another decision that I made while working on the feature is _not_ to change `randint` method calls where the object is known to be a `RandomState`. E.g. there are many tests that go like:

```python
random_state = RandomState(...)
i = random_state.randint(...)
```
or

```python
random_state = check_random_state(0)
i = random_state.randint(...)
```

Therefore, grepping through the repo for `randint` still reveals many direct calls, but unless I overlooked something, they should all be safe.

##### Caveats

It's almost impossible to have a complete test coverage for this feature. The reason is that even though we check all estimators that support `random_state`, we don't know if the code path that actually uses `random_state` is being taken or not, since it might depend on hyper-parameters. A similar argument applies to splitters and other functions.

user: BenjaminBossan
issue_23939: state: open
title: Sparse Bernoulli Naive Bayes
body: ### Describe the workflow you want to enable

The idea seems simple enough as described in this paper: https://arxiv.org/pdf/1905.09884.pdf

> Due to its linear complexity, naive Bayes classification remains an attractive su-
> pervised learning method, especially in very large-scale settings. We propose a
> sparse version of naive Bayes, which can be used for feature selection. This leads
> to a combinatorial maximum-likelihood problem, for which we provide an exact
> solution in the case of binary data, or a bound in the multinomial case. We prove
> that our bound becomes tight as the marginal contribution of additional features
> decreases. Both binary and multinomial sparse models are solvable in time almost
> linear in problem size, representing a very small extra relative cost compared to
> the classical naive Bayes. Numerical experiments on text data show that the naive
> Bayes feature selection method is as statistically effective as state-of-the-art feature
> selection methods such as recursive feature elimination, l1-penalized logistic re-
> gression and LASSO, while being orders of magnitude faster. For a large data set,
> having more than with 1.6 million training points and about 12 million features,
> and with a non-optimized CPU implementation, our sparse naive Bayes model can
> be trained in less than 15 seconds.

### Describe your proposed solution

Add a regularization parameter to the current implementation of BernoulliNB that allows to control sparsity of the solution similar to regularization strength or "C" in LogisticRegression with a L1 penalty.

Make the necessary adjustments to the algorithm.
user: carlosg-m
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23931: state: open
title: [WIP | First Contribution] add first pass at TimeSeriesInitialSplit
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
WIP #23923 


#### What does this implement/fix? Explain your changes.

Add functionality to do time series cross validation with an initial window.

#### Any other comments?

I have added the code, but I would like to add tests as well. I am just having difficulty setting up the test environment.

user: alexhallam
issue_23928: state: open
title: RFC SLEP006: verbose vs non-verbose declaration in meta-estimator
body: As the proposal and the implementation of meta-estimator routing (SLEP006) stands, if the user wants to use `sample_weight`, they need to be quite verbose in how they declare the estimators. Taking `AdaBoostClassifier` as an example, and imagining if `AdaBoostClassifier` would use the sub-estimator's `score` method, the user would have to write:

``` python
est = (
    AdaBoostClassifier(LogisticRegression().set_fit_request(sample_weight=True)
    .set_score_request(sample_weight=True))
    .fit(X, y, sample_weight=sw)
)
```

which is quite more verbose than the current code users need to write:

``` python
est = AdaBoostClassifier(LogisticRegression()).fit(X, y, sample_weight=sw)
```

There have been concerns about making users write quite verbose code in cases where the current pattern seems quite reasonable.

Without changing everything related to SLEP006, there are three paths we can take:

#### Option 1: Helper function
We can introduce helper functions to make the above code simpler. For instance, a `weighted` function could request `sample_weight` on all methods which accept `sample_weight` for a given estimator. Then the above code would look like:

``` python
est = AdaBoostClassifier(weighted(LogisticRegression())).fit(X, y, sample_weight=sw)
```

and if the sub-estimator is a pipeline:

``` python
est = AdaBoostClassifier(
    make_pipeline(weighted(StandardScaler()), weighted(LogisticRegression())))
).fit(X, y, sample_weight=sw)
```

Implementing `weighted` for a `Pipeline` (or other meta-estimators) would be tricky since `set_fit_request` is only available for consumers and not non-consumer routers; therefore the user needs to repeat the `weighted` call for all sub-estimators.

#### Option 2: Different meta-estimators
Have two classes of meta-estimators (or routers to be specific).

In this scenario, we divide meta-estimators into two classes, _simple_ and _complex_. Simple routers are the ones which simply forward `**kwargs` to sub-estimators, and by default the assume sub-estimators have requested those metadata. This simplifies the users' code and makes the existing code for simple meta-estimators to keep working, but it raises a few issues.

First is that there will be two classes of meta-estimators, and the user would need to know which estimator is of which class. It's also not clear what we should do if the user explicitly sets request values for metadata (we can probably respect those if present).

Another issue is that if a meta-estimator changes behavior, it needs to become a _complex_ meta-estimator if we want to keep backward compatibility for it. This doesn't seem like a good pattern.

#### Option 3: Keep as is
Do nothing, things are as is.

I'm in favor or option 1 because:
- with the helper function the user code doesn't look too verbose
- using metadata is not a beginner kinda thing and therefore this API is not hampering beginners' experience with the library
- it keeps consistency among meta-estimators/consumers

xref: https://github.com/scikit-learn/scikit-learn/pull/22986#pullrequestreview-1005144559

user: adrinjalali
issue_23923: state: open
title: TimeSeriesSplit Needs a version that allows users to specify initial time
body: ### Describe the workflow you want to enable

I would like to add a new `TimeSeriesSplit` [link](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html) called `TimeSeriesRollingOriginSplit`. In the current `TimeSeriesSplit` it is not possible for users to get an initial length.

I am happy to open a PR for this myself if it is approved.


### Describe your proposed solution


This is the proposed new `TimeSeriesRollingOriginSplit`
`class sklearn.model_selection.TimeSeriesRollingOriginSplit(initial = 5, test_size = 1, gap = 0)`

`initial`: the initial number of samples to use. it is often common to start with a larger training period.

`test_size`: Used to set the test_size

`gap`: Number of samples to exclude from the end of each train set before the test set.


### Describe alternatives you've considered, if relevant

I currently use `TimeSeriesSplit` in a loop, but exclude the initial splits until the train set is the size I want.

### Additional context

Again. I am happy to contribute this feature. 
user: alexhallam
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23920: state: open
title: RFC SLEP006: allow users to enable a "strict" mode in metadata routing
body: ref: https://github.com/scikit-learn/scikit-learn/pull/22986/files#r862344847

Along the way we've had discussions on whether we should raise if a metadata is requested but not provided.

Mirroring the existing behavior, SLEP006 and the implementation won't raise in the following case:

``` python
pipeline = make_pipeline(LogisticRegression().set_fit_request(sample_weight=True)).fit(X, y)
```

Should we allow a strict mode, either globally or on the estimator level, which would make the above code raise?

user: adrinjalali
issue_23901: state: open
title: Build time test issue on arm64 and ppc64el
body: ### Describe the bug

The Debian autobuilders catched a build time test error for the architectures [arm64](https://buildd.debian.org/status/fetch.php?pkg=scikit-learn&arch=arm64&ver=1.1.1-1&stamp=1653341247&raw=0) and [ppc64el](https://buildd.debian.org/status/fetch.php?pkg=scikit-learn&arch=ppc64el&ver=1.1.1-1&stamp=1653347532&raw=0).

This relates to issue #22503 but that was reported against version 1.0.2.  I created a new issue since it now links to the recent build logs of your latest version. 

### Steps/Code to Reproduce

If you have access to arm64 or ppc64el architectures try to build the code and run the test suite.

### Expected Results

See linked logs.

### Actual Results

If you browse the end of the logs I've linked above both architectures are ending up with the following test suite error:

    =================================== FAILURES ===================================
    ________ [doctest] sklearn.ensemble._weight_boosting.AdaBoostRegressor _________
    1026     Examples
    1027     --------
    1028     >>> from sklearn.ensemble import AdaBoostRegressor
    1029     >>> from sklearn.datasets import make_regression
    1030     >>> X, y = make_regression(n_features=4, n_informative=2,
    1031     ...                        random_state=0, shuffle=False)
    1032     >>> regr = AdaBoostRegressor(random_state=0, n_estimators=100)
    1033     >>> regr.fit(X, y)
    1034     AdaBoostRegressor(n_estimators=100, random_state=0)
    1035     >>> regr.predict([[0, 0, 0, 0]])
    Expected:
        array([4.7972...])
    Got:
        array([5.74049295])

The fact that the error is absolutely identical for am64 and ppc64el lets me assume that there is really an issue on non-amd64 64 bit architectures. 

### Versions

```shell
The packaged version ofscikit-learn which belongs to the linked autobuilder logs is 1.1.1
```

user: tillea
labels: arch:arm: ARM related issues
module:test-suite: everything related to our tests
issue_23897: state: open
title: Issue with affinity propagation and dtype np.float32
body: While working on https://github.com/scikit-learn/scikit-learn/pull/23838, it seems that the issue observed in https://github.com/scikit-learn/scikit-learn/issues/10832 was not solved since it breaks for different random seeds.

Therefore, we should revisit the original issue.
user: glemaitre
issue_23890: state: open
title: NearestCentroid not handling properly distance metrics other than Manhattan or Euclidean.
body: ### Describe the bug

Three issues, quite related.

* Using a metric that is not Manhattan or Euclidean triggers the following warning: `UserWarning: Averaging for metrics other than euclidean and manhattan not supported. The average is set to be the mean.`, but I believe an error message should be thrown instead. Indeed, computing the centroids as the average of the class samples is wrong for (I think) all distances except Euclidean, so it is misleading to actually do it even with a warning. Throwing an error here would fix the 2 following issues.

* Calling `.predict` on a `NearestCentroid` with metric `wminkowski`, `seuclidean` or `mahalanobis` throws an error, because those metrics need params that we can't pass to `NearestCentroid`. #23806 would fix the documentation, but maybe it should be nice to handle this properly in the code.

* Using any string as a `metric` (even a random string like `foobar`) does not throws an error when fitting, but does when calling `.predict`. Doing this with another classifier like `KNeighborsClassifier` throws an error at the fitting step.

If relevant I'd like to work on fixing those issues (new contributor here :)) I noticed a stalled PR (#19262) related to implementing `partial_fit` to `NearestCentroid`. Maybe I should start working on it first, as it seems to improve the code quality of `_nearest_centroid.py`, so it would be an interesting base to start with.

### Steps/Code to Reproduce

```python
#Reproduces the second issue mentionned

from sklearn.neighbors import NearestCentroid
from sklearn.datasets import load_iris
iris = load_iris()
X = iris.data
y = iris.target
nearest_centroid = NearestCentroid(metric="seuclidean")
nearest_centroid.fit(X,y)
nearest_centroid.predict(X)
```

### Expected Results

Fitting a `NearestCentroid` with a metric that is not Manhattan or Euclidean should trigger a proper error message.

### Actual Results

Error related to the second issue:
`ValueError: The 'V' parameter is required for the seuclidean metric when Y is passed.`

### Versions

```shell
System:
    python: 3.8.6 (default, Mar 29 2021, 18:42:37)  [Clang 12.0.0 (clang-1200.0.32.29)]
executable: /Users/valentinlaurent/.pyenv/versions/3.8.6/envs/lewagon/bin/python3.8
   machine: macOS-10.15.7-x86_64-i386-64bit

Python dependencies:
          pip: 22.1.1
   setuptools: 49.2.1
      sklearn: 1.0.2
        numpy: 1.19.5
        scipy: 1.6.2
       Cython: 0.29.26
       pandas: 1.1.3
   matplotlib: 3.4.1
       joblib: 1.1.0
threadpoolctl: 2.1.0

Built with OpenMP: True
```

user: Valentin-Laurent
issue_23886: state: open
title: Different inertia values even with random_state set
body: ### Describe the bug

Im trying to make my algorithm, which involves KMeans, reproducible, but even If I set random_state value I always get different inertia values. 
I'm using this [page](https://github.com/danielgribel/hg-means/blob/master/data/page.txt) dataset without the first line (dataset size).

### Steps/Code to Reproduce

```
from sklearn.cluster import KMeans
import numpy as np

points = np.genfromtxt('page.txt')
for l in range(10):
    kmeans = KMeans(n_clusters=30, n_init=1, random_state=42)
    kmeans.fit(points)

    centroids = kmeans.cluster_centers_
    print('{0:.20f}'.format(kmeans.inertia_))
```

### Expected Results

I expect to get printed ten time the same value

### Actual Results

1008436173.14048004150390625000
1008436173.14048004150390625000
1008436173.14047992229461669922
1008436173.14048004150390625000
1008436173.14047992229461669922
1008436173.14048004150390625000
1008436173.14047992229461669922
1008436173.14048004150390625000
1008436173.14048004150390625000
1008436173.14048004150390625000

### Versions

```shell
System:
    python: 3.9.12 (v3.9.12:b28265d7e6, Mar 23 2022, 18:22:40)  [Clang 13.0.0 (clang-1300.0.29.30)]
executable: /Users/user/PycharmProjects/MDE/venv/bin/python
   machine: macOS-12.4-arm64-arm-64bit
Python dependencies:
      sklearn: 1.1.0
          pip: 21.3.1
   setuptools: 60.2.0
        numpy: 1.22.4
        scipy: 1.8.0
       Cython: None
       pandas: 1.4.2
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0
Built with OpenMP: True
```

user: pzsette
issue_23885: state: open
title: check_estimator() not respecting custom estimator constraints via check_estimators_fit_returns_self()
body: ### Describe the bug

# check_estimators_fit_returns_self() does not pass with n_feature constraints
Building a custom estimator extending BaseEstimator with certain constraints such as:

## Example of constraints in the test.
```
X = check_array(
    X, 
    accept_sparse = False,
    accept_large_sparse = False,
    ensure_min_samples = 2,
    ensure_min_features = 3,
    force_all_finite = 'allow-nan')
```
The input and output of the estimator results in error thrown within `check_estimator`.

It appears that the tests run by `check_estimator` still check with input data with n_features = 2 for instance. Thus, the check_array with the above constraints will throw an error. This is not limited to the number of features but also whether force_all_finite allows nan or not. Some tests check with this value as True.

The test `check_estimators_fit_returns_self()` generates blobs with n_features = 2. Thus, this test will fail for any estimator that has feature_constraints != 2. 

See: https://github.com/scikit-learn/scikit-learn/blob/baf0ea25d/sklearn/utils/estimator_checks.py#L2581-L2595

## TL:DR
It appears that despite setting certain constraints of the input X,y. Such as `force_all_finite` and `ensure_min_features`. Some tests still use the default values for the `check_array` function and fail the check_estimator test due to this.

## The full error
<details>
<summary> Full error </summary>

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [5], in <cell line: 1>()
----> 1 check_estimator(model)

File ~/miniforge3/lib/python3.9/site-packages/sklearn/utils/estimator_checks.py:623, in check_estimator(estimator, generate_only, Estimator)
    621 for estimator, check in checks_generator():
    622     try:
--> 623         check(estimator)
    624     except SkipTest as exception:
    625         # SkipTest is thrown when pandas can't be imported, or by checks
    626         # that are in the xfail_checks tag
    627         warnings.warn(str(exception), SkipTestWarning)

File ~/miniforge3/lib/python3.9/site-packages/sklearn/utils/_testing.py:318, in _IgnoreWarnings.__call__.<locals>.wrapper(*args, **kwargs)
    316 with warnings.catch_warnings():
    317     warnings.simplefilter("ignore", self.category)
--> 318     return fn(*args, **kwargs)

File ~/miniforge3/lib/python3.9/site-packages/sklearn/utils/estimator_checks.py:2595, in check_estimators_fit_returns_self(name, estimator_orig, readonly_memmap)
   2592     X, y = create_memmap_backed_data([X, y])
   2594 set_random_state(estimator)
-> 2595 assert estimator.fit(X, y) is estimator

File ~/Documents/VOIDEV/tandem-riding-detection-service/service/source/models/physics/_base.py:42, in PhysicsRegressor.fit(self, X, y)
     20 """A fitting function which only checks the format of X and y.
     21 
     22 About the expected input data
   (...)
     38     Returns self.
     39 """
     40 self.n_features_in_ = 3
---> 42 X, y = check_X_y(
     43     X,
     44     y,
     45     accept_sparse = False,
     46     accept_large_sparse = False,
     47     ensure_min_samples = 2,
     48     ensure_min_features = 3,
     49     force_all_finite = True)
     51 #assert X.shape[-1] == 3, f"Expected 3 features. Received {X.shape[-1]}"
     53 return self

File ~/miniforge3/lib/python3.9/site-packages/sklearn/utils/validation.py:1074, in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
   1069         estimator_name = _check_estimator_name(estimator)
   1070     raise ValueError(
   1071         f"{estimator_name} requires y to be passed, but the target y is None"
   1072     )
-> 1074 X = check_array(
   1075     X,
   1076     accept_sparse=accept_sparse,
   1077     accept_large_sparse=accept_large_sparse,
   1078     dtype=dtype,
   1079     order=order,
   1080     copy=copy,
   1081     force_all_finite=force_all_finite,
   1082     ensure_2d=ensure_2d,
   1083     allow_nd=allow_nd,
   1084     ensure_min_samples=ensure_min_samples,
   1085     ensure_min_features=ensure_min_features,
   1086     estimator=estimator,
   1087     input_name="X",
   1088 )
   1090 y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
   1092 check_consistent_length(X, y)

File ~/miniforge3/lib/python3.9/site-packages/sklearn/utils/validation.py:918, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
    916     n_features = array.shape[1]
    917     if n_features < ensure_min_features:
--> 918         raise ValueError(
    919             "Found array with %d feature(s) (shape=%s) while"
    920             " a minimum of %d is required%s."
    921             % (n_features, array.shape, ensure_min_features, context)
    922         )
    924 if copy and np.may_share_memory(array, array_orig):
    925     array = np.array(array, dtype=dtype, order=order)

ValueError: Found array with 2 feature(s) (shape=(21, 2)) while a minimum of 3 is required.
```
</details>

## Reproduction
Let's create a very basic estimator which only takes 3 features. Then run check_estimator on this.

```
class DummyEstimator(BaseEstimator):
    def __init__(self):
        pass

    def fit(self, X, y=None):
        X, y = check_X_y(
            X,
            y,
            accept_sparse = False,
            accept_large_sparse = False,
            ensure_min_features = 3,
            force_all_finite = 'allow-nan')
        return self

    def predict(self, X):
        X  = check_array(
            X,
            accept_sparse = False,
            accept_large_sparse = False,
            ensure_min_features = 3,
            force_all_finite = 'allow-nan')
        
        # very dumb example but it does not matter
        y_pred = X[:,0] + X[:,1] + X[:,3]
        
        y_pred  = check_array(
            y_pred,
            accept_sparse = False,
            accept_large_sparse = False,
            ensure_min_features = 3,
            ensure_2d = False,
            force_all_finite = 'allow-nan')
        
        return y_pred
```

Error produced:
```ValueError: Found array with 2 feature(s) (shape=(21, 2)) while a minimum of 3 is required.```

### Steps/Code to Reproduce

Run a check_estimator on an instance of this dummy estimator

```
class DummyEstimator(BaseEstimator):
    def __init__(self):
        pass

    def fit(self, X, y=None):
        X, y = check_X_y(
            X,
            y,
            accept_sparse = False,
            accept_large_sparse = False,
            ensure_min_features = 3,
            force_all_finite = 'allow-nan')
        return self

    def predict(self, X):
        X  = check_array(
            X,
            accept_sparse = False,
            accept_large_sparse = False,
            ensure_min_features = 3,
            force_all_finite = 'allow-nan')
        
        # very dumb example but it does not matter
        y_pred = X[:,0] + X[:,1] + X[:,3]
        
        y_pred  = check_array(
            y_pred,
            accept_sparse = False,
            accept_large_sparse = False,
            ensure_min_features = 3,
            ensure_2d = False,
            force_all_finite = 'allow-nan')
        
        return y_pred
```

### Expected Results

Passing all tests.

### Actual Results

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [5], in <cell line: 1>()
----> 1 check_estimator(DummyEstimator())

File ~/miniforge3/lib/python3.9/site-packages/sklearn/utils/estimator_checks.py:623, in check_estimator(estimator, generate_only, Estimator)
    621 for estimator, check in checks_generator():
    622     try:
--> 623         check(estimator)
    624     except SkipTest as exception:
    625         # SkipTest is thrown when pandas can't be imported, or by checks
    626         # that are in the xfail_checks tag
    627         warnings.warn(str(exception), SkipTestWarning)

File ~/miniforge3/lib/python3.9/site-packages/sklearn/utils/_testing.py:318, in _IgnoreWarnings.__call__.<locals>.wrapper(*args, **kwargs)
    316 with warnings.catch_warnings():
    317     warnings.simplefilter("ignore", self.category)
--> 318     return fn(*args, **kwargs)

File ~/miniforge3/lib/python3.9/site-packages/sklearn/utils/estimator_checks.py:2595, in check_estimators_fit_returns_self(name, estimator_orig, readonly_memmap)
   2592     X, y = create_memmap_backed_data([X, y])
   2594 set_random_state(estimator)
-> 2595 assert estimator.fit(X, y) is estimator

File ~/Documents/VOIDEV/tandem-riding-detection-service/service/source/models/physics/_base.py:15, in DummyEstimator.fit(self, X, y)
     14 def fit(self, X, y=None):
---> 15     X, y = check_X_y(
     16         X,
     17         y,
     18         accept_sparse = False,
     19         accept_large_sparse = False,
     20         ensure_min_features = 3,
     21         force_all_finite = 'allow-nan')
     22     return self

File ~/miniforge3/lib/python3.9/site-packages/sklearn/utils/validation.py:1074, in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
   1069         estimator_name = _check_estimator_name(estimator)
   1070     raise ValueError(
   1071         f"{estimator_name} requires y to be passed, but the target y is None"
   1072     )
-> 1074 X = check_array(
   1075     X,
   1076     accept_sparse=accept_sparse,
   1077     accept_large_sparse=accept_large_sparse,
   1078     dtype=dtype,
   1079     order=order,
   1080     copy=copy,
   1081     force_all_finite=force_all_finite,
   1082     ensure_2d=ensure_2d,
   1083     allow_nd=allow_nd,
   1084     ensure_min_samples=ensure_min_samples,
   1085     ensure_min_features=ensure_min_features,
   1086     estimator=estimator,
   1087     input_name="X",
   1088 )
   1090 y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
   1092 check_consistent_length(X, y)

File ~/miniforge3/lib/python3.9/site-packages/sklearn/utils/validation.py:918, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
    916     n_features = array.shape[1]
    917     if n_features < ensure_min_features:
--> 918         raise ValueError(
    919             "Found array with %d feature(s) (shape=%s) while"
    920             " a minimum of %d is required%s."
    921             % (n_features, array.shape, ensure_min_features, context)
    922         )
    924 if copy and np.may_share_memory(array, array_orig):
    925     array = np.array(array, dtype=dtype, order=order)

ValueError: Found array with 2 feature(s) (shape=(21, 2)) while a minimum of 3 is required.
```

### Versions

```shell
System:
    python: 3.9.7 (default, Sep 16 2021, 23:53:23)  [Clang 12.0.0 ]
executable: /Users/alexandervialabellander/miniforge3/bin/python
   machine: macOS-12.2.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.1.1
          pip: 21.2.4
   setuptools: 58.0.4
        numpy: 1.22.3
        scipy: 1.7.3
       Cython: None
       pandas: 1.3.5
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /Users/alexandervialabellander/miniforge3/lib/python3.9/site-packages/sklearn/.dylibs/libomp.dylib
        version: None
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /Users/alexandervialabellander/miniforge3/lib/python3.9/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.18
threading_layer: pthreads
   architecture: armv8
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /Users/alexandervialabellander/miniforge3/lib/libopenblasp-r0.3.17.dylib
        version: 0.3.17
threading_layer: pthreads
   architecture: armv8
    num_threads: 8
```

user: AlexVialaBellander
labels: module:test-suite: everything related to our tests
issue_23876: state: open
title: [MRG] PERF Significant performance improvements in Partial Least Squares (PLS) Regression
body: ## Description of Changes

#### Motivating Summary

This PR dramatically improves the speed of PLS regression by implementing the Dayal-MacGregor modified kernel algorithm ([Dayal-MacGregor 1997](https://analyticalsciencejournals.onlinelibrary.wiley.com/doi/10.1002/%28SICI%291099-128X%28199701%2911%3A1%3C73%3A%3AAID-CEM435%3E3.0.CO%3B2-%23)). This has significant performance improvements over the current default, NIPALS, without sacrificing accuracy or numerical stability. 

The Dayal-MacGregor algorithm accurately computes the weight and rotation matrices for the maximal covariance projection(s) of _X_ without explicitly calculating the _X_ score matrix and without deflating the _Y_ target(s) at all. Thus it is (much) faster than NIPALS in the base case of univariate _Y_, and becomes asymptotically faster as the number of _Y_ targets increases. Testing shows speed improvements of over 40% in the base case, and rising to over 95% when _Y_ is multivariate.

In other words, this PR makes PLS regression 2x - 20x faster for equivalent results. The paper contains detailed numerical stability characteristics and accuracy results which are confirmed by my empirical tests below.

#### Specific Changes

- the parameter constraints on the `algorithm` parameter for the `_PLS` base class were modified so that a user can select the new algorithm via `algorithm="dayalmacgregor"` or `algorithm="kernel"` - see comments below, I'm not attached to having both arguments refer to the new algorithm, but I've implemented it this way for now
- the `PLSRegression` child class was modified to allow selection of the `algorithm` parameter during instantiation (previously a user could not select the algorithm, and the regression child class implicitly took the default of the base class). this argument defaults to `algorithm="nipals"` for backwards compatibility
- the `fit` method was augmented to include the implementation of Dayal-MacGregor, within a conditional branch on the value of the `algorithm` attribute
- Added a new test to cover the new algorithm

## Performance & Accuracy Results

#### Small X (20 x 10) and Univariate Y (20 x 1) with 5 Components

NIPALS: 344 µs ± 4.77 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Dayal-MacGregor: 198 µs ± 3.87 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

The output predictions agree to 6 decimal places.

_43% faster_

#### Small X (20 x 10) and Multivariate Y (20 x 2) with 5 Components

NIPALS: 700 µs ± 13.7 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Dayal-MacGregor: 268 µs ± 4.92 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

The output predictions are equal to 4 decimal places.

_62% faster_

#### Moderate X (200 x 100) and Univariate Y (200 x 1) with 50 Components

NIPALS: 32.6 ms ± 2.16 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Dayal-MacGregor: 15 ms ± 470 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

The output predictions agree to 6 decimal places.

_54% faster_

#### Moderate X (200 x 100) and Multivariate Y (200 x 2) with 50 Components

NIPALS: 226 ms ± 21.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Dayal-MacGregor: 15.8 ms ± 280 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

The output predictions are equal to 6 decimal places.

_94% faster_

#### Moderate X (200 x 100) and Multivariate Y (200 x 3) with 50 Components

NIPALS: 346 ms ± 29.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Dayal-MacGregor: 17 ms ± 1.48 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

The output predictions are equal to 5 decimal places.

_95% faster_

#### Large X (2000 x 1000) and Univariate Y (2000 x 1) with 100 Components

NIPALS: 859 ms ± 27.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Dayal-MacGregor: 271 ms ± 20.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

The output predictions are equal to 6 decimal places.

_59% faster_

#### Large X (2000 x 1000) and Multivariate Y (2000 x 2) with 100 Components

NIPALS: 5.71 s ± 390 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Dayal-MacGregor: 233 ms ± 10.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

The output predictions are equal to 6 decimal places.

_96% faster_

#### Large X (2000 x 1000) and Multivariate Y (2000 x 3) with 100 Components

NIPALS: 6.43 s ± 398 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Dayal-MacGregor: 236 ms ± 13.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

The output predictions agree to 6 decimal places.

_97% faster_

## Testing

#### Dev Environment & Architecture

My testing environment is as follows:

- Hardware: MacBook Pro (16-inc, 2021) with Apple M1 Max Chip and 64GB RAM
- OS: macOS Monterey v12.4
- Python: 3.10.5
- Installer/Package Manager: Anaconda3 (for Apple Silicon/arm64)
- Building Compiler: llvm-openmp

I don't think this will materially impact the results, but wanted to note it.

#### Testing Script

I put together a performance and accuracy testing script.

After running `conda create -n sklearn-dev -c conda-forge python numpy scipy cython joblib threadpoolctl pytest compilers llvm-openmp`, `conda activate sklearn-dev` and `pip install --verbose --no-build-isolation --editable .` in the directory, the following testing script should validate the foregoing performance and accuracy assessment. It randomly generates an _m_ x _n_ matrix _X_ and _m_ x _n_targets_ matrix _Y_, and then fits PLS regression models estimated with each of the NIPALS and Dayal-MacGregor algorithms, then assesses performance and accuracy.

```
import numpy as np
from numpy.testing import assert_array_almost_equal
from sklearn.cross_decomposition import PLSRegression

#%% Config
m = 1000
n = 200
n_targets = 2

X = np.random.rand(m, n)
y = np.random.rand(m, n_targets)
n_components = 50
scale = True

####
# Performance Testing
####

#%% NIPALS
%%timeit
nipals = PLSRegression(n_components=n_components, scale=scale, algorithm='nipals')
nipals.fit(X=X, Y=y)
nipals.predict(X)

#%% DayalMacGregor Kernel
%%timeit
dayal = PLSRegression(n_components=n_components, scale=scale, algorithm='dayalmacgregor')
dayal.fit(X=X, Y=y)
dayal.predict(X)

#%%
####
# Accuracy/Stability Testing
####

control = PLSRegression(n_components=n_components, scale=scale, algorithm='nipals')
test = PLSRegression(n_components=n_components, scale=scale, algorithm='dayalmacgregor')
control.fit(X, y)
test.fit(X, y)
assert_array_almost_equal(test.predict(X), control.predict(X), decimal=4)
```

## Comments

- I kept NIPALS as the default algorithm. If this PR is accepted, we might want to discuss whether or not NIPALS should remain the default given that Dayal-MacGregor is strictly superior for `PLSRegression`. Maybe keep NIPALS as the default, mention the new algorithm prominently in documentation and issue a warning that the new algorithm will become the default in a release or two?

- Given that Dayal-MacGregor doesn't deflate the _Y_ target(s), there are no `y_scores_`, `y_weights_` and `y_rotations_` class attributes. This means that Dayal-MacGregor is only implemented for PLS Regression (not PLS Canonical), and a `ValueError` check is implemented to that effect. Moreover, the `PLSRegression` class documentation was edited to reflect the fact that these three attributes are only given for NIPALS/SVD.

- Should this algorithm be available via both `algorithm="dayalmacgregor"` and `algorithm="kernel"`, or is it better to choose one? I currently have both implemented. I can see arguments either way but I'm not strongly opinionated. The Dayal-MacGregor algorithm is a kernel algorithm, but it's not the only one that exists in the literature (e.g. SIMPLS is also a kernel algorithm, albeit numerically unstable).
user: fractionalhare
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23840: state: open
title: page on the website that provides direction to users on how to reach out
body: from @amueller 
>Hey! Is there a page on the website that summarizes how to reach out and how we prefer people reach out? So far I only see collections of links. I kind of expected the "community" menu item to tell me how to engage with the community, but it currently links to the blog.

>The most concise summary seems to be in https://scikit-learn.org/dev/faq.html#what-s-the-best-way-to-get-help-on-scikit-learn-usage but it's not really the right question and it's a hard place to find, and outdated.

---
Copying text from Discord

This Discord server is intended for communication between the scikit-learn maintainers and community contributors during **Office Hours** and **Sprints**. If you have questions, this is our general workflow:

* **GitHub Discussions** 
     (usage questions such as methodological):  
     https://github.com/scikit-learn/scikit-learn/discussions
* **StackOverflow**
     (programming questions):
     with tag ‘scikit-learn’
* **GitHub Issue Tracker**
     (bug reports): 
     https://github.com/scikit-learn/scikit-learn/issues
     Please do not ask usage questions on the issue tracker.
* **Discord server**
     (current pull requests):
     Please post any specific PR-related questions on your PR, and you can share a link to your PR on this server during office hours.

Reference on asking questions:
http://matthewrocklin.com/blog/2019/02/28/slack-github

**Calendar **for Office Hours:
https://blog.scikit-learn.org/calendar/
user: reshamas
labels: Easy: Well-defined and straightforward way to resolve
issue_23837: state: open
title: Unclear selection of LOF threshold
body: In https://github.com/scikit-learn/scikit-learn/blob/baf0ea25d6dd034403370fea552b21a6776bef18/sklearn/neighbors/_lof.py#L103-L104 the documentation states that if the parameter 'contamination' is set to 'auto', the threshold for LOF will be calculated as outlined in the original paper.

But here https://github.com/scikit-learn/scikit-learn/blob/baf0ea25d6dd034403370fea552b21a6776bef18/sklearn/neighbors/_lof.py#L147-L148 and here https://github.com/scikit-learn/scikit-learn/blob/baf0ea25d6dd034403370fea552b21a6776bef18/sklearn/neighbors/_lof.py#L310  we can see that the threshold is set to 1.5 in all cases.
The original paper (https://doi.org/10.1145/335191.335388) never mentions the "magic number" 1.5, which makes me wonder where it comes from. 

I'd be glad if you could clarify this. If someone is able to provide a reference for the value of 1.5 I'll gladly update the documentation.

EDIT: The original paper mentions the number 1.5, but only in an example (Sec. 7.3). I don't think they meant to suggest that there is anything special about the number 1.5, it might even be possible that they settled on this number because of space constraints in the paper.
user: gutjuri
labels: Needs Investigation: Issue requires investigation
issue_23824: state: open
title: FEA Add strategy isotonic to calibration curve
body: #### Reference Issues/PRs
Fixes #23132.

#### What does this implement/fix? Explain your changes.
This PR adds `strategy="isotonic"` to `calibration_curve` and `CalibrationDisplay`.

#### Any other comments?
Reliability diagrams with (PAV algorithm) isotonic regression is the CORP approach of (https://doi.org/10.1073/pnas.2016191118).

user: lorentzenchr
issue_23817: state: open
title: Add a LearningCurveSplitter to test sample efficiency
body: ### Describe the workflow you want to enable

In applications of machine learning the error of the model typically follows a power law with the number of training examples. In the ML for material science and molecules community there is lots of interest in using such plots to asses model performance however as of yet there is no standard splitting tool to make such splits. A standard splitter would improve the reproducibility and uptake of such studies.

### Describe your proposed solution

A learning-curve splitter that returns a series of training sets of various sizes alongside a fixed test set. Kwargs would control whether the splitting size was logarithmic, how many splits, whether to take multiple splits for each training set size, etc

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

![image](https://user-images.githubusercontent.com/26601751/176975465-1b36d087-6f17-4d06-a2e7-eb3ffcccc2a7.png)
This is an example of the type of plot used to compare models. https://www.nature.com/articles/s41467-020-18556-9
user: CompRhys
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23801: state: open
title: Add `ref_line_kwargs` to CalibrationDisplay
body: ### Describe the workflow you want to enable

**I would like to style the `CalibrationDisplay` reference line by passing styling arguments  via a kwargs object.**

If I need the reference line colored `'white'`, I would like to enable the following workflow:
```python
fig, ax = plt.subplots()
CalibrationDisplay.from_estimator(model, X_test, y_test, ax=ax , ref_line_kwargs={'color':'white'})
```

### Describe your proposed solution

#### 1) Add a `ref_line_kwargs` parameter to the `CalibrationDisplay.plot` method. ([Line 1092](https://github.com/scikit-learn/scikit-learn/blob/baf0ea25d6dd034403370fea552b21a6776bef18/sklearn/calibration.py#L1092))
```python
def plot(self, *, ax=None, name=None, ref_line=True, ref_line_kwargs={}, **kwargs):
```

#### 2) Pass the `ref_line_kwargs` object into `ax.plot`. ([Line 1139](https://github.com/scikit-learn/scikit-learn/blob/baf0ea25d6dd034403370fea552b21a6776bef18/sklearn/calibration.py#L1139))
```python
ax.plot([0, 1], [0, 1], "k:", label=ref_line_label, **ref_line_kwargs)
```

### Describe alternatives you've considered, if relevant

#### There are two alternatives that allow styling the reference line
1. Add a reference line manually and set `ref_line=False`
2. Add a reference line manually and set the label for the line to `'Perfectly calibrated'`

#### 1)
```python
fig, ax = plt.subplots()
ax.plot([0, 1], [0, 1], "k:", color='white')
CalibrationDisplay.from_estimator(model, X_test, y_test, ref_line=False, ax=ax)
```

#### 2)
```python
fig, ax = plt.subplots()
ax.plot([0, 1], [0, 1], "k:", label='Perfectly calibrated', color='white')
CalibrationDisplay.from_estimator(model, X_test, y_test, ax=ax)
```

### Additional context

For my current project, to generate a white reference line, I found myself copying and pasting the source code used for plotting the reference line by sklearn, and then adding the styling edits I needed. 

The primary benefit of this change would be the ability to style all of the plotted data without needing to interact with the matplotlib api. 

There appears to be an active pull request for improving the calibration plotting tools, and given the simplicity of this change, and if others agree with the request, #23748 seems like a natural place for this update. 
user: joelsewhere
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23780: state: open
title: ENH Add option `n_splits='walk_forward'` in `TimeSeriesSplit`
body: #### Reference Issues/PRs
Fixes #22523 

#### What does this implement/fix? Explain your changes.
This pull request adds a requested feature in the Issue #22523: "walk_forward" for the `TimeSeriesSplit` class. 

Based on what I understand about this feature, here are some important details:
* Walking forward is defined by the first element of the first train set starting from 0 and being incremented by 1 in the subsequent train and test indices while adhering to the max train size and test size values. For example: 
  ```
  # Walk Forward Example where max_train_size = 10, test_size = 2, x.shape[0] = 15 
  # n_splits is automatically computed to 3 in this case 
  TRAIN:  [0 1 2 3 4 5 6 7 8] TEST:  [ 9 10]
  TRAIN:  [ 1  2  3  4  5  6  7  8  9 10] TEST:  [11 12]
  TRAIN:  [ 3  4  5  6  7  8  9 10 11 12] TEST:  [13 14]
  ```
* To make waking forward possible the `n_splits` value is set to `walk_forward`: 
  `cv = TimeSeriesSplit(n_splits="walk_forward", x_shape=x.shape[0], max_train_size=10 ,test_size=2)`. 
  However, as this parameter was originally designed only to receive integer values I had to make some modifications to the   constructor of the `_BaseKFold()` super class to make this possible. Also, the `x.shape[0]` value from `x = np.arange(15)` is needed to be passed into the constructor in order to automatically compute values for `n_splits` parameter

* Once the `n_splits` value is set to `walk_forward`, the `find_walk_forward_n_splits_value()` method is called in order to calculate an appropriate integer value for `n_splits` so that walking forward is possible. In the case of multiple potential `n_splits` values, the first element in an array containing those values will be returned: 
  ```      
  if len(n_splits_arrays_first_element_zero) > 0:
              return n_splits_arrays_first_element_zero[0]
  ```

#### Any other comments?
I have regression tested manually with the original functionality and it seems to work fine. 
There are two tests in the `sklearn/model_selection/_split.py` file that are failing(I'm looking into why now)

Let me know if this potential solution works, needs modifications or if you have any feedback.

user: ShehanAT
issue_23779: state: open
title: `OneVsOneClassifier` does not accept custom input types
body: ### Describe the bug

Due to the additional validation in #6626, `OneVsOneClassifier` cannot be used with custom types that work like arrays but cannot be converted to arrays. In my case I am the maintainer of [scikit-fda](https://fda.readthedocs.io/en/latest/), a functional data library that attempts to be compatible with scikit-learn. The metaestimators `OneVsRestClassifier` and `OneVsOneClassifier` should be trivially compatible with our data. Currently `OneVsRestClassifier` works fine, while `OneVsOneClassifier` doesn't.

I think that scikit-learn has sometimes very aggressive validation that makes it difficult to extend it for custom objects, as in this case.

### Steps/Code to Reproduce

I have no example using only scikit-learn, as you need custom types and classifiers / transformers to trigger it
The following code is adapted from https://fda.readthedocs.io/en/latest/auto_tutorial/plot_skfda_sklearn.html#multiclass-and-multioutput-classification-utilities
Note that the code works if you replace OneVsOneClassifier with OneVsRestClassifier, as the later does not have that aggressive validation.

```python
import skfda
import skfda.preprocessing.dim_reduction.variable_selection as vs
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.multiclass import OneVsOneClassifier

X, y = skfda.datasets.fetch_phoneme(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

pipeline = Pipeline([
    ("dim_reduction", vs.RKHSVariableSelection(n_features_to_select=3)),
    ("classifier", SVC()),
])

multiclass = OneVsOneClassifier(pipeline)

multiclass.fit(X_train, y_train)
multiclass.score(X_test, y_test)
```

### Expected Results

The result of the classification.

### Actual Results

```
TypeError                                 Traceback (most recent call last)
TypeError: float() argument must be a string or a number, not 'FDataGrid'

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-2-66e12b97fe3e> in <module>
     17 multiclass = OneVsOneClassifier(pipeline)
     18 
---> 19 multiclass.fit(X_train, y_train)
     20 multiclass.score(X_test, y_test)

~/Programas/Utilidades/Lenguajes/miniconda3/envs/fda38/lib/python3.8/site-packages/sklearn/multiclass.py in fit(self, X, y)
    726         """
    727         # We need to validate the data because we do a safe_indexing later.
--> 728         X, y = self._validate_data(
    729             X, y, accept_sparse=["csr", "csc"], force_all_finite=False
    730         )

~/Programas/Utilidades/Lenguajes/miniconda3/envs/fda38/lib/python3.8/site-packages/sklearn/base.py in _validate_data(self, X, y, reset, validate_separately, **check_params)
    579                 y = check_array(y, **check_y_params)
    580             else:
--> 581                 X, y = check_X_y(X, y, **check_params)
    582             out = X, y
    583 

~/Programas/Utilidades/Lenguajes/miniconda3/envs/fda38/lib/python3.8/site-packages/sklearn/utils/validation.py in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
    962         raise ValueError("y cannot be None")
    963 
--> 964     X = check_array(
    965         X,
    966         accept_sparse=accept_sparse,

~/Programas/Utilidades/Lenguajes/miniconda3/envs/fda38/lib/python3.8/site-packages/sklearn/utils/validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
    744                     array = array.astype(dtype, casting="unsafe", copy=False)
    745                 else:
--> 746                     array = np.asarray(array, order=order, dtype=dtype)
    747             except ComplexWarning as complex_warning:
    748                 raise ValueError(

ValueError: setting an array element with a sequence.

```

### Versions

```shell
System:
    python: 3.8.0 | packaged by conda-forge | (default, Nov 22 2019, 19:11:38)  [GCC 7.3.0]
executable: /home/carlos/Programas/Utilidades/Lenguajes/miniconda3/envs/fda38/bin/python3.8
   machine: Linux-5.4.0-92-generic-x86_64-with-glibc2.10

Python dependencies:
          pip: 21.3.1
   setuptools: 60.5.0
      sklearn: 1.0.2
        numpy: 1.22.0
        scipy: 1.7.3
       Cython: 0.29.26
       pandas: 1.4.0
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.0.0

Built with OpenMP: True
```

user: vnmabus
issue_23767: state: open
title: Additive Score/Metric decomposition into Miscalibration, Discrimination and Uncertainty
body: ### Describe the workflow you want to enable

### Proposition
I'd like to decompose scores (at least the ones from consistent scoring functions for identifiable functionals) into meaningful additive components:
- MSC ≥ 0 miscalibration (for Brier score also know as reliability term)
- DSC ≥ 0 discrimination (for Brier score also know as resolution term)
- UNC ≥ 0 uncertainty or entropy

as (lower is better) `score_loss = MSC - DSC + UNC`, see [1, 2, 3, 4]. An implementation in R is available at [5].

#### For classification with Brier score
```python
from sklearn.metrics import brier_score_loss, score_decompose

model.fit(X_train, y_train)
y_pred = model.predict(y_test)
msc, dsc, unc = score_decompose(y_test, y_pred, brier_score_loss)

assert msc - dsc + unc = brier_score_loss(y_test, y_pred)
```
#### For quantile regression with pinball loss
```python
rom sklearn.metrics import mean_pinball_loss, score_decompose

alpha = 0.8  # the 80% quantile
model.set_params(quantile=alpha).fit(X_train, y_train)
y_pred = model.predict(y_test)
msc, dsc, unc = score_decompose(y_test, y_pred, brier_score_loss, alpha=alpha)

assert msc - dsc + unc = mean_pinball_loss(y_test, y_pred, alpha=alpha)
```

#### References
[1] Pohle (2020) https://arxiv.org/abs/2005.01835
[2] Dimitriadis, Gneiting, Jordan (2020) https://arxiv.org/abs/2008.03033
[3] Gneiting & Resin (2021) https://arxiv.org/abs/2108.03210
[4] Fissler, Lorentzen & Mayer (2022) https://arxiv.org/abs/2202.12780
[5] https://github.com/aijordan/reliabilitydiag

### Describe your proposed solution

From [2]:
>  While there is a consensus on the character and intuitive interpretation of the decomposition terms, their exact form remains subject to debate, despite a half century quest in the wake of Murphy’s [1973] Brier score decomposition. In particular, Murphy’s decomposition is exact in the discrete case, but fails to be exact under continuous forecasts, which has prompted the development of increasingly complex types of decompositions (...).

> In the extant literature, it has been assumed implicitly or explicitly that the calibrated and reference forecasts can be chosen at researchers’ discretion (...). We argue otherwise and contend that the calibrated forecasts ought to be the PAV-(re)calibrated probabilities, as displayed in the CORP reliability diagram, whereas the reference forecast r ought to be the marginal event frequency (...). We refer to the resulting decomposition as the CORP score decomposition, which enjoys the following properties:
• MCB ≥ 0 with equality if the original forecast is calibrated.
• DSC ≥ 0 with equality if the PAV-calibrated forecast is constant.
• The decomposition is exact.

> Perhaps surprisingly, the PAV algorithm and its appealing properties generalize from prob-
abilistic classifiers to mean, quantile, and expectile assessments for real-valued outcomes

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

 #23132 is related and might be a good first step towards `score_decomposition`.

Solving this issue will also solve #21774.
user: lorentzenchr
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23750: state: open
title: fetch_lfw_pairs issue
body: ### Describe the bug

On two different machines for exactly same code, I am getting different result.



### Steps/Code to Reproduce

```py
from sklearn.datasets import fetch_lfw_pairs
from sklearn.utils import resample

X = fetch_lfw_pairs(
    subset="test",
    # funneled=False,
    # slice_=(slice(0, 250), slice(0, 250)),
    resize=1,
    color=True
  )

n_samples = 1
imgs, y = resample(X.pairs, X.target, n_samples=n_samples, random_state=101)

print(imgs[0][0])
```

### Expected Results

Both on both machines should be the same.

### Actual Results

Result on Machine 1:

```
array([[[ 77.,  77.,  77.],
        [ 94.,  88.,  76.],
        [107., 101.,  77.],
        ...,
        [159., 124.,  97.],
        [141., 110.,  84.],
        [136., 109.,  82.]],

       [[ 78.,  78.,  80.],
        [108., 101.,  91.],
        [128., 121.,  99.],
        ...,
        [159., 124.,  97.],
        [151., 120.,  94.],
        [149., 124.,  98.]],

       [[ 82.,  80.,  85.],
        [111., 104.,  96.],
        [130., 122., 103.],
        ...,
        [163., 131., 105.],
        [157., 129., 104.],
        [159., 138., 110.]],

       ...,

       [[ 27.,  27.,  27.],
        [ 27.,  27.,  27.],
        [ 24.,  24.,  24.],
        ...,
        [151., 172., 235.],
        [142., 163., 228.],
        [128., 150., 210.]],

       [[ 26.,  24.,  25.],
        [ 33.,  33.,  33.],
        [ 30.,  29.,  27.],
        ...,
        [150., 171., 236.],
        [136., 157., 222.],
        [112., 133., 194.]],

       [[ 31.,  30.,  26.],
        [ 36.,  35.,  33.],
        [ 37.,  36.,  34.],
        ...,
        [146., 167., 230.],
        [124., 145., 208.],
        [ 93., 112., 171.]]], dtype=float32)
```

Result on Machine 2:
```
array([[[0.        , 0.00392157, 0.        ],
        [0.00392157, 0.01176471, 0.04705882],
        [0.01960784, 0.07450981, 0.16470589],
        ...,
        [0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.00392157]],

       [[0.        , 0.        , 0.        ],
        [0.00392157, 0.01176471, 0.05098039],
        [0.01960784, 0.07843138, 0.16862746],
        ...,
        [0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.00392157]],

       [[0.        , 0.00392157, 0.        ],
        [0.00392157, 0.01568628, 0.05490196],
        [0.01568628, 0.07450981, 0.16470589],
        ...,
        [0.        , 0.        , 0.00392157],
        [0.        , 0.00392157, 0.        ],
        [0.        , 0.00392157, 0.        ]],

       ...,

       [[0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ],
        ...,
        [0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ]],

       [[0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ],
        ...,
        [0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ]],

       [[0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ],
        ...,
        [0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ],
        [0.        , 0.        , 0.        ]]], dtype=float32)

```

### Versions

```shell
Machine 1:

System:
    python: 3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0]
executable: /home/anks/miniconda3/envs/stealth/bin/python
   machine: Linux-5.17.9-1-MANJARO-x86_64-with-glibc2.10

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 1.0.2
        numpy: 1.21.2
        scipy: 1.7.3
       Cython: 0.29.30
       pandas: 1.3.0
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

Machine 2:

System:
    python: 3.8.5 | packaged by conda-forge | (default, Sep 24 2020, 16:55:52)  [GCC 7.5.0]
executable: /home/aimluser/miniconda3/envs/application_env/bin/python3.8
   machine: Linux-4.4.0-042stab145.3-x86_64-with-glibc2.10

Python dependencies:
          pip: 22.1.2
   setuptools: 62.1.0
      sklearn: 1.0.2
        numpy: 1.22.3
        scipy: 1.8.0
       Cython: None
       pandas: 1.3.0
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

user: AnkS4
labels: Needs Investigation: Issue requires investigation
issue_23748: state: open
title: [WIP] Enhance calibration plots
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Will close #23709 

#### What does this implement/fix? Explain your changes.
Add an histogram to the calibration plot. For now, works only when only one classifier per plot (no staking).

#### Any other comments?
Work in progress

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: aperezlebel
issue_23746: state: open
title: [WIP] DOC Explain missing value mechanisms
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Addresses task 3 of #21967.

#### What does this implement/fix? Explain your changes.
Add a section to the "Imputation of missing values" doc to explain the missing value mechanisms.

#### Any other comments?
Work in progress

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: aperezlebel
issue_23732: state: open
title: Add `n_jobs` and `random_state` to global config
body: ### Describe the workflow you want to enable

I would like to be able to set `n_jobs=-1` in one place and have this take effect in any function with an `n_jobs` parameter. Same for `random_state`.

Perhaps there are other parameters that fit the theme: "if a user sets this in one instance, they _probably_ want to set it in all instances".

### Describe your proposed solution

Expand the accepted parameters of [sklearn.set_config](https://scikit-learn.org/stable/modules/generated/sklearn.set_config.html#sklearn.set_config), update all functions to fall back to the config value if the parameter isn't passed.

This would require changing a default of `None` to a sentinel (ala `_NoValue` in NumPy), to allow a user to override the global config with the value `None`, while still allowing the code to detect if the argument was passed.

Rudimentary mockup:
```py
_sentinel = object()

config = {}


def resolve_arg_value(arg_name, passed_value, default_value):
    if passed_value is not _sentinel:
        return passed_value
    
    if arg_name in config:
        return config[arg_name]
    
    return default_value


def do_something(random_state=_sentinel):
    random_state = resolve_arg_value("random_state", random_state, None)
    print(f"{random_state!s}")


do_something('Hi')  # 'Hi'
do_something()  # None
do_something(None)  # None

config['random_state'] = 77

do_something('Hi')  # 'Hi'
do_something()  # 77
do_something(None)  # None

config.pop('random_state')

do_something('Hi')  # 'Hi'
do_something()  # None
do_something(None)  # None
```

I'll admit, having to add something like `resolve_arg_value("random_state", random_state, None)` to tons of functions sounds painful, but I think for the user, being able to set and forget a random state/other params, potentially based on an environment variable would be nice.
```py
sklearn.set_config(
    random_state=None if os.environ["PROD"] else 0,
    n_jobs=-1,
)
```


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I see that the tests get a global random seed. https://github.com/scikit-learn/scikit-learn/pull/22749
user: davidgilbertson
labels: Needs Decision - API: Requires decision regarding API
issue_23728: state: open
title: [Meta] consistently break ties randomly in scikit-learn estimators (with random_state)
body: Some estimators have arbitrary ways to break ties:

- k-nearest neighbors with data points lying on a uniform 1d grid (see #23667 for the BallTree for instance);
- feature splits in histogram gradient boosted trees with redundant features.
- possibly other models (please feel free to edit this list or suggest missing estimators in the comments).

If the tie breaking logic is deterministic, then it might introduce a non-controllable bias in a datascience pipeline. For instance when analyzing the feature importance of an histogram gradient boosting model (via permutations or SHAP values), the first feature of a group of redundant features would always deterministically be picked up by the model and could lead a naive datascientist to believe that the other features of the group are not as predictive.

Note that this is not the case for our traditional `DecisionTreeClassifier/Regressor` / `RandomForestClassifier/Regressor` and extra trees because they all do [feature shuffling](https://github.com/scikit-learn/scikit-learn/blob/c217527af5744b9d0db8761c1e3667552312e5e7/sklearn/tree/_splitter.pyx#L322) (controllable by `random_state`) by default even when `max_features == 1.0`. This makes it easy to conduct the same study many times with different seeds to see if the results are an artifact of an arbitrary tie breaking or not.

user: ogrisel
labels: Needs Decision: Requires decision
issue_23727: state: open
title: [RFC] WASM / pyodide as a (somewhat) officially supported platform for scikit-learn
body: We started having bug reports (at least one indirect, in real life report at a conference: #23707) from users of scikit-learn in WASM environment (e.g. pyodide / jupyterlite, pyscript...).

Shall we invest effort in setting CI tooling to properly test and maybe even handle packaging of scikit-learn to target that platform?

It's very likely that not all of scikit-learn will work out of the box, but with proper tooling in place we could maintain a public list of modules that have all their tests that pass and maybe a list of modules that required so patches to handle graceful degradation to target this platform (e.g. number of parallel worker threads with n_jobs).

@rth put some interesting info in the following comment on how to run the tests:

- https://github.com/scikit-learn/scikit-learn/issues/23707#issuecomment-1163053666

Pros:
- WASM is likely to be a very popular target platform, at least for education (can directly teach Python programming and ML concepts without having to teach how to install packages from the command line first).

Cons:
- test execution is probably much slower that on our regular CI targets;
- need to maintain a list of known issues / limitations;
- more packaging, release process will be even more complicated;
- SciPy is quite heavily patched because there is no working Fortran compiler on that platform  (that might change soon with lfotran) so it relies on a semi-hackish Fortran to C transpilation step that introduces additional complexity.
user: ogrisel
labels: Needs Investigation: Issue requires investigation
issue_23715: state: open
title: t-SNE score for generalized embedding method
body: ### Describe the workflow you want to enable

t-SNE is not incremental and it is very slow to perform on streaming data. However, in my research, I use the t-SNE score (i.e. the KL divergence between the `P` and `Q` distributions to evaluate the "goodness" of a generalized embedding method. I propose adding the t-SNE score that takes in the embeddings and the domain data and returns a metric that may be used to compare between embedding approaches. 

### Describe your proposed solution

Under `sklearn/manifold/_t_sne.py`, add the following function:
```py
def t_sne_score(
        perplexity : float = 30.0,
        metric : str = 'euclidean',
        squared : bool = True,
        n_jobs : int = 4,
        domain,
        embedding,
    ):
    """
        Comparative metric that allows to 
        asses the effectiveness of a generalized
        embedding method given the domain variable
        and the embedded variable. Computes the cost of 
        t-SNE for an embedding that may not be t-SNE.
        
        domain : NDArray is a 2D array that describes the data
                    to be embedded
        embedding: NDArray is a 2D array of shape 
                  (domain.shape[0], latent_space_dim) that describes 
                  the output of the embedding 
    """
    assert domain.shape[0] == embedding.shape[0], "Sample size between 
                                            domain variable and embedding
                                            must be the same"
    latent_space_dim = embedding.shape[1]
    degrees_of_freedom = max(embedding.shape[1] - 1, 1)
    distances = pairwise_distances(X = domain,
                                metric = metric,
                                squared = squared,
                                n_jobs = n_jobs)
    P = _t_sne._joint_probabilities(distances, perplexity, False)
    dist = pdist(embedding, 'sqeuclidean')
    dist /= degrees_of_freedom
    dist += 1.0
    dist **=(degrees_of_freedom + 1.0) / -2.0
    Q = np.maximum(dist / (2.0 * np.sum(dist)), MACHINE_EPSILON)
    return 2.0 * np.dot(P, np.log(np.maximum(P, MACHINE_EPSILON) / Q))
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: sarahaguasvivas
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23709: state: open
title: Enhance calibration plots
body: ### Describe the workflow you want to enable

I would like to enhance calibration plots by adding these new aspects:
1. Plot bins as vertical lines in the background.
2. Equal aspect on both axis to have a square figure and enable easier visual comparisons.
3. Changing xlabel from "Mean predicted probability" to "Mean confidence scores" or "Confidence scores" or similar, to avoid confusion between scores output by classifiers and true posterior probabilities.
4. Integrate a visualization of the density to see where the mass is on the calibration curve. This is important as a calibration curve can appear very bad in some part whereas it can be due to a low number of samples in this part, and actually have a good Expected Calibration Error. Several choices are possible as described in the proposed solution section.
5. Edit: add support for multiclass calibration plot by either looking at the class-wise calibration (considering one of the classes as the positive one) or the top-label problem (binarizing the multiclass problem with the prediction taken as the argmax of the confidences, a positive label iff the prediction is right and an associated confidence given by the maximum of the confidences). These are 2 of the 3 definitions of calibration in the multiclass setting.

### Describe your proposed solution

To address the previous points, I propose to:
1. Update `CalibrationDisplay`.
2. Idem
3. Idem
4. Several choices are possible here. Either adding an histogram on each edge as in the figure below. Or adding a single histogram on the confidence scores axis (either on the edge or integrated in the background of the calibration plot) since the density on the "fraction of positive" axis is less informative as the one on the confidence scores axis. This last option might be better. Also, this feature might be restricted to a single calibration curve only as it might become illegible with several classifiers.
5. Edit: This could be done by adapting the `_get_response` to allow y_pred.shape[1] > 2 and either choosing one of the K class as the positive one, or the top-label one.

Should this change be approved, I would be willing to work on it.

Example with histograms on the edges:
<img src="https://user-images.githubusercontent.com/33580936/174766358-56fc2afa-5452-4318-b0cb-18caf1b2a004.jpg" alt="drawing" width="400"/>

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: aperezlebel
issue_23676: state: open
title: Inside SearchCV objects, enable `return_estimators`, similar to `cross_validate`
body: ### Describe the workflow you want to enable

I am running GridSearch/RandomizedSearchCV and need to have access to the underlying estimators to perform some inquiries on the fitted objects based on their performances. However, there is no way of getting the fitted estimators.


### Describe your proposed solution

Add a `return_estimators=False` argument to the Grid/RandomizedSearchCV objects with a warning that one should use caution if their search space is large, since some models can be nontrivial size.

I would be interested in adding a PR if this is an acceptable feature request.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This came up in context of trying to investigate tree sizes based on their optimal performances fitted using SearchCV. 

cc: @ogri
user: adam2392
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23670: state: open
title: BUG unpenalized GLM with LBFGS solver does not yield minimum norm solution with fit_intercept=True
body: ### Summary
While unpenalized GLMs like `PoissonRegressor(alpha=0, fit_intercept=False)` yield a minimum norm solution on wide data, `PoissonRegressor(alpha=0, fit_intercept=True)` does not.

Detected in #23314.

#### Minimal Reproducible Example
```python
import numpy as np
from numpy.testing import assert_allclose
import pytest
from sklearn.datasets import make_low_rank_matrix
from sklearn.linear_model import PoissonRegressor
from sklearn.linear_model._linear_loss import LinearModelLoss


alpha = 0  # unpenalized, even tiny values like 1e-10 does not help.
fit_intercept = True  # if set to False, all tests pass.
model = PoissonRegressor(
    alpha = alpha,
    fit_intercept=fit_intercept,
    tol=1e-12,
    max_iter=1000,
)

# same as glm_dataset in test_glm.py for "wide"
n_samples, n_features = 4, 12
k = n_samples
rng = np.random.RandomState(42)
X = make_low_rank_matrix(
    n_samples=n_samples,
    n_features=n_features,
    effective_rank=k,
    tail_strength=0.1,
    random_state=rng,
)
X[:, -1] = 1  # last columns acts as intercept
U, s, Vt = np.linalg.svd(X, full_matrices=False)

raw_prediction = rng.uniform(low=-3, high=3, size=n_samples)
# minimum norm solution min ||w||_2 such that raw_prediction = X w:
# w = X'(XX')^-1 raw_prediction = V s^-1 U' raw_prediction
coef_unpenalized = Vt.T @ np.diag(1 / s) @ U.T @ raw_prediction

linear_loss = LinearModelLoss(base_loss=model._get_loss(), fit_intercept=True)
sw = np.full(shape=n_samples, fill_value=1 / n_samples)
y = linear_loss.base_loss.link.inverse(raw_prediction)

if fit_intercept:
    X = X[:, :-1]  # remove intercept
    intercept = coef_unpenalized[-1]
    coef = coef_unpenalized[:-1]
else:
    intercept = 0
    coef = coef_unpenalized

model.fit(X, y)

# This is true. So we have a solution.
assert_allclose(model.predict(X), y, rtol=1e-6)

# But it is NOT the minimum norm solution
norm_solution = np.linalg.norm(np.r_[intercept, coef])
norm_model = np.linalg.norm(np.r_[model.intercept_, model.coef_])
print(f"norm_solution = {norm_solution}")
print(f"norm_model    = {norm_model}")
assert norm_model == pytest.approx(norm_solution, rel=1e-4)
```
Output
```
norm_solution = 2.494710738642327
norm_model    = 2.5295744931923707
AssertionError
```

The strange thing is that setting `fit_intercept=False`, norms and coefficients are equal, i.e. the minimum norm solution is returned by the LBFGS solver.
On top, even setting a tiny positive penalty like `alpha=1e-10` does not give the minimum norm solution.
user: lorentzenchr
issue_23667: state: open
title: BallTree.query returns inconsistent indices between scikit-learn versions 0.24.1 and 1.1.1
body: ### Describe the bug

For the same data, `BallTree.query` can return different nearest neighbour indices if these neighbours are equally distanced from a point depending on scikit-learn version. Specifically, this can be observed in 0.24.1 vs 1.1.1.  

### Steps/Code to Reproduce

```
import numpy as np
from sklearn.neighbors import BallTree

np.random.seed(61)

X = np.random.randint(0, 3, size=(10,2))  # generated dataset

tree = BallTree(X, 4, 'hamming')
distances, indices = tree.query(X, 3, return_distance=True, dualtree=False, breadth_first=False, sort_results=True)

for i, (point, distances, indices) in enumerate(zip(X, distances, indices)):
    print(f'index {i}: datapoint {point} distances: {[round(dist,2) for dist in distances]} indices: {indices}')

```


### Expected Results

scikit-learn versions 0.24.1 and 1.1.1 would be expected to consistently show the same nearest neighbour indices whenever multiple neighbours are equally distanced from a data point

### Actual Results

scikit-learn v. 0.24.1:

<img width="549" alt="scikit-0 24 1" src="https://user-images.githubusercontent.com/21366061/174240375-d7215db5-6137-4ed6-a0d3-647d41761e19.png">

scikit-learn v. 1.1.1:

<img width="549" alt="scikit-1 1 1" src="https://user-images.githubusercontent.com/21366061/174240420-e4684662-26b1-4808-ae72-8e4ccff8e751.png">




### Versions

```shell
System:
  python: 3.9.7 (v3.9.7:1016ef3790, Aug 30 2021, 16:39:15) [Clang 6.0 (clang-600.0.57)]
executable: /Users/korigo/Downloads/scikit-0.24/bin/python3
  machine: macOS-10.16-x86_64-i386-64bit
Python dependencies:
     pip: 22.0.4
  setuptools: 62.1.0
   sklearn: 0.24.1
    numpy: 1.22.4
    scipy: 1.8.1
    Cython: None
    pandas: None
  matplotlib: None
    joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

System:
  python: 3.9.7 (v3.9.7:1016ef3790, Aug 30 2021, 16:39:15) [Clang 6.0 (clang-600.0.57)]
executable: /Users/korigo/Downloads/scikit-1.1.1/bin/python3
  machine: macOS-10.16-x86_64-i386-64bit
Python dependencies:
   sklearn: 1.1.1
     pip: 22.0.4
  setuptools: 62.1.0
    numpy: 1.22.4
    scipy: 1.8.1
    Cython: None
    pandas: None
  matplotlib: None
    joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

user: eeghor
issue_23653: state: open
title: [RFC] Support for int64 indexed SciPy sparse matrices in Cython code
body: At the moment we do not have systematic support for very large sparse matrices in our Cython code. That would be useful when the data is passed as a sparse matrix with more than ~2e9 columns or non-zero values.

The purpose of this issue is to link:

- reference all related issues in scikit-learn.
- decide if we want to have some uniform support guarantees or not
- decide if we need centralized Cython tooling (e.g. type declarations, tempita conventions) to add support for such matrices.

### Related issues and PRs (feel free to update this list):

For polynomial feature expansion (quite popular request):

- #16803
- #17554
- #19676 

Other models with open issues:

- #11355
- #11356
- #18090
- #18403

Other Cython estimators that could also be updated:

- neighbors models (k-NN and radius-based models) 
  - related issues not just about this problem: #23604
- k-means & variants
- Feature Hasher / Hashing Vectorizer (`sklearn/feature_extraction/_hashing_fast.pyx`)

The following PR will introduce a scikit-learn transformer that can output `int64` indexed sparse matrices (even if it's input is `int32` indexed).

- #23731 

### Helpful Python snippet

SciPy decides to use the int32 or int64 dtype depending on the dimensions of the matrix and on the number of stored non-zero elements. Here is a quick way to generate a CSR matrix that requires int64-typed `.indices` and `.indptr` attributes:

```python
>>> from scipy.sparse import csr_matrix
>>> import numpy as np
>>>
>>> X = csr_matrix(([1.0], [np.iinfo(np.int32).max + 1], [0, 1]))
>>> X
<1x2147483649 sparse matrix of type '<class 'numpy.float64'>'
        with 1 stored elements in Compressed Sparse Row format>
>>> X.indices
array([2147483648])
>>> X.indices.dtype
dtype('int64')
>>> X.indptr.dtype
dtype('int64')
```
user: ogrisel
labels: Needs Decision: Requires decision
issue_23644: state: open
title: FIX Adopted more direct `dtype` restraint in preperation for `NEP50`
body: #### Reference Issues/PRs
In response to [NEP 50](https://github.com/numpy/numpy/pull/21626)

#### What does this implement/fix? Explain your changes.
Currently when building with the (mostly ready) NEP 50 [implementation](https://github.com/numpy/numpy/pull/21626), the tests fail for `sklearn.linear_model.tests.test_bayes::test_dtype_match`. This PR makes some small changes to ensure compliance with NEP 50 style dtypes.

#### Any other comments?

user: Micky774
issue_23616: state: open
title: WIP ENH Added `auto` option to `FastICA.whiten_solver`
body: #### Reference Issues/PRs
Follow-up to #22527

#### What does this implement/fix? Explain your changes.
Added `auto` option to `FastICA.whiten_solver` along w/ tests, beginning deprecation to adopt `auto` as default value for `whiten_solver`.

#### Any other comments?
For justification of `>50x` heuristic in choosing solver, please see [this gist](https://gist.github.com/Micky774/e69fcce0b865c9b6c4f35598f0932249) which contains the generating benchmark script as well as a copy of my results.
user: Micky774
issue_23603: state: open
title: [WIP] Enable multi-output voting regression
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Partially fixes #18289.

#### What does this implement/fix? Explain your changes.
Enables multi-output regression with VotingRegressor.

#### Any other comments?
I am aware that another PR exists in regards to this (#19223), but it seems to be stale and I think the regression part is quite a simple fix, which is why I created this PR.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: wullli
issue_23589: state: open
title: DOC Replace `chi2` with `f_classif` in feature selection examples
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Fixes #17286.

#### What does this implement/fix? Explain your changes.

In this PR, I will fix the usage of the `chi2` function. The `chi2` should be applied to the count of categorical features, but some examples use `chi2` function to estimate the continuous variable features. It may confuse users. I think that the `f_classif` is a preferable choice for the continuous features of classification tasks. 

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


In "examples/compose/plot_compare_reduction.py", I added the following changes to remove some `UserWarning.

```diff
- "reduce_dim": [PCA(iterated_power=7), NMF()],
+ "reduce_dim": [PCA(iterated_power=7), NMF(init="nndsvda", max_iter=1000)],
```        

There still exist the following warnings, but I think these are irrelevant to the PR.

```
scikit-learn/sklearn/feature_selection/_univariate_selection.py:110: UserWarning: Features [ 0 32 39] are constant.
  warnings.warn("Features %s are constant." % constant_features_idx, UserWarning)
scikit-learn/sklearn/feature_selection/_univariate_selection.py:111: RuntimeWarning: invalid value encountered in true_divide
  f = msb / msw
```

user: i-aki-y
issue_23574: state: open
title: Segmentation Fault in KMeans on OSX
body: ### Describe the bug

Hi when I run this code

```
import numpy as np      
from sklearn.cluster import KMeans

X_train = np.random.RandomState(0).random((10, 2))
kmeans = KMeans(n_clusters=3, random_state=0).fit(X_train)
```

EDIT by @ogrisel: inserted `.RandomState(0)` in the above snippet to make the reproducer deterministic.

I get this error

`zsh: segmentation fault  python debugging.py`

This can't be a memory error, it's only a 10x2 matrix, any idea what's wrong?

Python 3.10.4
Anaconda installation
>>> sklearn.__version__
'1.0.2'
>>> numpy.__version__
'1.22.3'


### Steps/Code to Reproduce


```
import numpy as np      
from sklearn.cluster import KMeans

X_train = np.random.random((10, 2))
kmeans = KMeans(n_clusters=3, random_state=0).fit(X_train)
```

### Expected Results

KMeans clusters

### Actual Results

`zsh: segmentation fault  python debugging.py`

This can't be a memory error, it's only a 10x2 matrix, any idea what's wrong?


### Versions

```shell
Python 3.10.4
Anaconda installation
>>> sklearn.__version__
'1.0.2'
>>> numpy.__version__
'1.22.3'
```

user: EoinKenny
labels: Needs Investigation: Issue requires investigation
issue_23568: state: open
title: Allow subclasses to pass through check_array()
body: ### Describe the workflow you want to enable

I have a subclass of `numpy.ndarray` which keeps track of axis labels: which axes represent time, spatial directions, independent samples, and response variables.  I have a `transform()` step in my pipeline that runs
`x = check_array(x, accept_sparse=("csr", "csc"), ...)`.
It's nice that this allows me to handle both numpy arrays and sparse inputs .  However, this [line](https://github.com/scikit-learn/scikit-learn/blob/b73ec3c3b39f2fcda37f943f47cf90b8f46253c4/sklearn/utils/validation.py#L847) of `scipy.utils.validation.check_array` downcasts anything that isn't a sparse array to a numpy array.  This is because `numpy.asarray` does not allow interception via `__array_ufunc__` or `__array_function__`, so subclasses cannot dictate how to handle its behavior.

### Describe your proposed solution

`isinstance` check to see if `array` is already a numpy array.  Not sure exactly how to enforce `order` and `dtype`, however.

### Describe alternatives you've considered, if relevant

User code instead repeats validation code from check_array.  In my case, that means importing the private function `_ensure_sparse_format()` and skipping or copypasting the other validation checks.

### Additional context

_No response_
user: Jacob-Stevens-Haas
issue_23567: state: open
title: Include print counter for HalvingGridSearch in verbose=3 mode
body: ### Describe the workflow you want to enable

Dear all,
when using HalvingGridSearch it is very likely, that (especially the first iteration) it contains 100's to 1000's of grid points. While monitoring in the verbose mode can be nice to get an idea about parameter importance, the only way to get an idea about the progress is to 'count rows'. 

### Describe your proposed solution

To get a better idea on the progress it would be nice to add the counter in the verbose = 3 output:
e.g. instead of 
"[CV 1/3] END C=5, class_weight=None, degree=2, gamma=1, kernel=rbf;, score=(train=-0.050, test=-0.201) total time=  13.5s"

"[**Iter 0; Fit Number 1234 of 4321;** CV 1/3] END C=5, class_weight=None, degree=2, gamma=1, kernel=rbf;, score=(train=-0.050, test=-0.201) total time=  13.5s"

Both, Iter and total number of fits in the current iteration are provided at the beginning of each iteration and thus would require only minor changes.


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: JB91451
issue_23551: state: open
title: DOC Improvements to developer documentation
body: Improvements to developer documentation:

- In `advanced_installation.rst`:
    - added windows case when creating a virtual env without conda
    - added recommendation to check the folder before running the build command
- In `contributing.rst`:
    - added reference to GitHub guide to generate SSH key (that's necessary to run some commands) .

user: Eschivo
issue_23550: state: open
title: `distance_threshold` not respected in `AgglomerativeClustering` with sparse `connectivity`
body: ### Describe the bug

When passing a sparse `connectivity` to `AgglomerativeClustering` constrained by a given `distance_threshold`, the current implementation may return a clustering not respecting this constraint.

This is at least true for the 'complete' and 'average' linkage criteria. I found the origin of the problem in [`linkage_tree`](https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/cluster/_agglomerative.py#L384), because I saw the only moment the inter-point distances were calculated was here:

https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/cluster/_agglomerative.py#L565-L568

So the only distance info we have is for connected points. Then, for `linkage='complete'`, in the situation where you have clustered together two neighbours A and B, and in the next step you consider aggregating C, which is B's neighbour but not A's, if `d(A, C) > distance_threshold` but at the same time `d(B, C)` is the minimum remaining distance (at the top of the heap), then C will be aggregated with these two.

Now I'm not sure at all on how to fix this properly. One option is to compute all pairwise distances in the connection matrix `A` when `A` and the heap `inertia` are initialised:

 https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/cluster/_agglomerative.py#L596-L604

or to somehow compute the ones needed on the fly when two nodes with different neighbourhoods are merged. Both solutions seem quite expensive though.

To give you some context, I encountered this while trying to do a spatial aggregation of administrative boundaries (England's LSOAs), in such a way that an indicator (let's say average income) does not vary too much within the aggregated regions. In this kind of geographic context, this is a rather common task, the [Ward variant](https://pysal.org/spopt/generated/spopt.region.WardSpatial.html#spopt.region.WardSpatial) is even implemented in the `pysal` package.

### Steps/Code to Reproduce

```python
import numpy as np
import pandas as pd
from sklearn.feature_extraction.image import grid_to_graph
from sklearn.cluster import AgglomerativeClustering

rng = np.random.RandomState(0)
n_samples = 100
conn = np.ones([n_samples, n_samples], dtype=bool)
# Add some random sparsity, 70% at most
masked = (rng.random((n_samples, 70)) * n_samples).astype(int)
for i, nbh in enumerate(masked):
    conn[i, nbh] = False
    conn[nbh, i] = False
X = rng.randn(n_samples, 1)
clustering = AgglomerativeClustering(
    n_clusters=None,
    distance_threshold=1,
    connectivity=conn,
    linkage='complete',
    affinity='l1',
)
clustering.fit(X)
df = pd.DataFrame({'labels': clustering.labels_, 'x': X.flatten()})
clustered_df = df.groupby("labels")["x"].agg(["min", "max"])

# This should be < distance_threshold
print((clustered_df['max'] - clustered_df['min']).max())
# 1.20248995732655
```

### Expected Results

A float lower than 1, meaning the linkage criterion was respected.

### Actual Results

1.20248995732655

### Versions

```shell
System:
    python: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:56:21)  [GCC 10.3.0]
executable: /home/thomaslouf/.conda/envs/ses-ling/bin/python
   machine: Linux-4.15.0-144-generic-x86_64-with-glibc2.27

Python dependencies:
      sklearn: 1.1.1
          pip: 22.1.2
   setuptools: 62.3.2
        numpy: 1.22.4
        scipy: 1.8.1
       Cython: None
       pandas: 1.4.2
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/thomaslouf/.conda/envs/ses-ling/lib/libopenblasp-r0.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: Bulldozer
    num_threads: 32

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/thomaslouf/.conda/envs/ses-ling/lib/libgomp.so.1.0.0
        version: None
    num_threads: 1
```

user: TLouf
labels: Needs Investigation: Issue requires investigation
issue_23533: state: open
title: RFECV race condition on estimator
body: In RFECV, at
https://github.com/scikit-learn/scikit-learn/blob/fb3ed90fb501a755ce2938fb566bd0f6e2235054/sklearn/feature_selection/_rfe.py#L723-L726
the estimator is passed as-is to the fit function. Since `fit` modifies the object without copying, this is prone to race conditions (see example below).

Contrast this to BaseSearchCV, where the estimator is properly cloned:
https://github.com/scikit-learn/scikit-learn/blob/fb3ed90fb501a755ce2938fb566bd0f6e2235054/sklearn/model_selection/_search.py#L823-L833

---

<details>
<summary>On my system, with parameter `n_jobs=-1`, I got the following error:</summary>

```
5 fits failed with the following error:
Traceback (most recent call last):
  File ".../site-packages/sklearn/model_selection/_validation.py", line 686, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File ".../site-packages/sklearn/feature_selection/_rfe.py", line 723, in fit
    scores = parallel(
  File ".../site-packages/joblib/parallel.py", line 1056, in __call__
    self.retrieve()
  File ".../site-packages/joblib/parallel.py", line 935, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File ".../site-packages/joblib/_parallel_backends.py", line 595, in __call__
    return self.func(*args, **kwargs)
  File ".../site-packages/joblib/parallel.py", line 262, in __call__
    return [func(*args, **kwargs)
  File ".../site-packages/joblib/parallel.py", line 262, in <listcomp>
    return [func(*args, **kwargs)
  File ".../site-packages/sklearn/utils/fixes.py", line 117, in __call__
    return self.function(*args, **kwargs)
  File ".../site-packages/sklearn/feature_selection/_rfe.py", line 37, in _rfe_single_fit
    return rfe._fit(
  File ".../site-packages/sklearn/feature_selection/_rfe.py", line 327, in _fit
    self.scores_.append(step_score(self.estimator_, features))
  File ".../site-packages/sklearn/feature_selection/_rfe.py", line 40, in <lambda>
    lambda estimator, features: _score(
  File ".../site-packages/sklearn/model_selection/_validation.py", line 767, in _score
    scores = scorer(estimator, X_test, y_test)
  File ".../site-packages/sklearn/metrics/_scorer.py", line 219, in __call__
    return self._score(
  File ".../site-packages/sklearn/metrics/_scorer.py", line 261, in _score
    y_pred = method_caller(estimator, "predict", X)
  File ".../site-packages/sklearn/metrics/_scorer.py", line 71, in _cached_call
    return getattr(estimator, method)(*args, **kwargs)
  File ".../site-packages/sklearn/ensemble/_forest.py", line 835, in predict
    return self.classes_.take(np.argmax(proba, axis=1), axis=0)
AttributeError: 'list' object has no attribute 'take'
```
</details>

It is generated from the following snippet:
```py
  rf = RandomForestClassifier()
  rfecv = RFECV(rf, scoring='accuracy', n_jobs=-1)
  rfecv.fit(X_train, y_train)
```

The error appears to happen because `n_outputs_` is not constant between runs. The error does not happen without parallelism.
user: cpiber
issue_23522: state: open
title: Bayesian Ridge Regression
body: ### Describe the bug

Using the Bayesian Ridge Regression with MultiOutputRegression(n_jobs=-1) consumes all the memory on the computer and the process dies. (Tested with 128G Ram + 128G Swap)

### Steps/Code to Reproduce

model = MultiOutputRegressor(sklearn.linear_model.BayesianRidge(), n_jobs=-1)
model = model.fit(train_lr, train_hr)
pred = model.predict(test_lr)

### Expected Results

No memory leak.

### Actual Results

Memory leak occurs

### Versions

```shell
tested on
scikit-learn==1.0.2
scikit-learn==0.24.2
```

user: sekomer
labels: Needs Investigation: Issue requires investigation
issue_23518: state: open
title: Fix resize segfault
body: The following code segment should be adjusted:
```python
self.capacity = node_ndarray.shape[0]
if self._resize_c(self.capacity) != 0:
```
It should instead be just `if self._resize_c(node_ndarray.shape[0]) != 0:`. Allow me to explain why:

In the `resize_c` function, the initial check is `if capacity == self.capacity and self.nodes != NULL:` . If we are editing an existing tree (which is probably bad practice, but it has several use cases so it might as well be supported for people that know what they're doing right?), the nodes are not NULL. This means that, because we pass `self.capacity` as an argument, it will evaluate to true and thus return 0 without resizing.

If the replacement I suggest is implemented, the resize will be executed and - as an effect of the resize function - the capacity will still be updated, but at the right moment.

Currently trying to load the state of one (large) tree into the state of another (small) tree will result in a segfault on certain operations.

https://github.com/scikit-learn/scikit-learn/blob/8ea2997dfcd72126cdd6793a2c5293406407b1dc/sklearn/tree/_tree.pyx#L699
user: Cesar48
labels: Needs Investigation: Issue requires investigation
issue_23450: state: open
title: Add neighbors algorithm based on NSW graphs
body: ### Describe the workflow you want to enable

Dear Scikit-Learn maintainers, good afternoon! 

I would like to add the algorithm to do the approximate nearest neighbors search to the sklearn.neighbors module.
The algorithm may enhance performance in the case of high-dimensional data space, starting from data dimension D > 50.
It will be the alternative to the existing [KDTree](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KDTree.html#sklearn.neighbors.KDTree) and [BallTree](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.BallTree.html#sklearn.neighbors.BallTree) methods.

The desired utilization can be as follows:
```
from sklearn.neighbors import NSWGraph
import numpy as np
rng = np.random.RandomState(0)
X = rng.random_sample((50, 128))  
nswgraph = NSWGraph()
nswgraph.build(X)
dist, ind = tree.query(X[:1], k=3)  
```
### Describe your proposed solution

The method is based on [Navigable small world graphs](http://www.scholarpedia.org/article/Small-world_network) (NSW graphs) that tends to demonstrate better performance in the high-dimensional data space [1] (D > 50).  This data structure is a graph which satisfies the property that the shortest path between two random vertices increases proportionally to the logarithm of the graph size [2].


### Describe alternatives you've considered, if relevant

The Scikit-Learn alternatives of the proposed method are the KDTree and BallTree algorithms that give a performance advantage in low-dimensional data but become inefficient as the dimensionality grows (suffer from [the curse of dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality)) [3].

### Additional context

Despite the created issue with feature request, you can find the actual implementation proposed as [the pull request](https://github.com/scikit-learn-contrib/scikit-learn-extra/pull/143) to the
 scikit-learn-extra project.

________
References

[1] Malkov, Y., Ponomarenko, A., Logvinov, A., & Krylov, V. (2014). 
Approximate nearest neighbor algorithm based on navigable small world graphs.
Information Systems, 45, 61-68.
[2] Sandberg, Oskar, and Ian Clarke. "The evolution of navigable small-world networks." arXiv preprint cs/0607025 (2006).
[3] Scikit-Learn User's guide: Nearest Neighbor Algorithms
    https://scikit-learn.org/stable/modules/neighbors.html#nearest-neighbor-algorithms
user: LeoSvalov
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23427: state: open
title: Poor performance of KNeighborsClassifier for sklearn version >=0.24.2
body: ### Describe the bug

Hi. I used **sklearn.neighbors.KNeighborsClassifier** in my program but I noticed that its runtime increased when adopting higher versions of  sklearn.

As shown in the following experiment results, this model performs best if we use sklearn <0.24.2.

**My question is that why does such a library version perform better? Are there any issues in newer versions?**

The detailed information is as follows:
Runtime | Memory | Version
-- | -- | --
**451.2495478** | **2337.8830499649** | **1.0.1**
**441.4340003** | **2338.7937555313** | **0.24.2**
26.25025550 | 414.1812286377 | 0.23.2
26.42619310 | 408.6705312729 | 0.22.1
26.42619310 | 408.6052923203 | 0.22
31.0837617 | 409.0471181870 | 0.21.3
29.0283316 | 408.5676021576 | 0.20.3
22.6652508 | 409.7873554230 | 0.19.2

My program only used the apis of  KNeighborsClassifier provided by sklearn.
```Python
knn = KNeighborsClassifier(n_neighbors = 3)
```


### Steps/Code to Reproduce

I create a minor program that used KNeighborsClassifier and you can test it on [colab](https://colab.research.google.com/drive/1GU9Hw8jKObcO5a54-7cNXAA-t9BJuKFi?usp=sharing). The runtime is still higher when adopting the newer sklearn version on this example.

[train.csv](https://drive.google.com/file/d/1f77gSB472ouhZ5bK0ipocBVa3SweGKLc/view?usp=sharing)

Using the above `train.csv` and the following snippet:

```python
# %%
# %%
import pandas as pd

train_data = pd.read_csv("train.csv")
total = train_data.isnull().sum().sort_values(ascending=False)
percent_1 = train_data.isnull().sum() / train_data.isnull().count() * 100
percent_2 = (round(percent_1, 1)).sort_values(ascending=False)
missing_data = pd.concat([total, percent_2], axis=1, keys=["Total", "%"])
cor = train_data.corr()
cor_target = abs(cor["target"])
relevant_features = cor_target[cor_target > 0.5]
s = train_data.dtypes == "object"
train_data_cat_var = list(s[s].index)
train_data.drop(
    ["nom_5", "nom_6", "nom_7", "nom_8", "nom_9", "ord_3", "ord_4", "ord_5"],
    axis=1,
    inplace=True,
)
train_data_cat_var = [
    ele
    for ele in train_data_cat_var
    if ele
    not in ["nom_5", "nom_6", "nom_7", "nom_8", "nom_9", "ord_3", "ord_4", "ord_5"]
]
final_train_data = pd.get_dummies(
    train_data, columns=train_data_cat_var, drop_first=True
)
features = final_train_data.drop(["target"], axis=1).columns
target = final_train_data["target"]

# %%
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    final_train_data[features], target, test_size=0.2
)

# %%
from sklearn.neighbors import KNeighborsClassifier

# %%
%%time
knn = KNeighborsClassifier(n_neighbors=3, algorithm="brute")
knn.fit(X_train, y_train)
Y_pred_knn = knn.predict(X_test)
```


![image-20220520110041914](https://user-images.githubusercontent.com/26241319/169444004-c50c2e22-489c-4ce9-8771-99bec82693bc.png)


### Expected Results

Similar runtime for the different sklearn versions or better performance for higher sklearn versions.

### Actual Results

According to the above experiment results, when sklearn version is 0.24.2 or 1.0.1, the performance of the model is worse.

### Versions


I test my program with sklearn 1.0.1, 0.24.2, 0.23.2, 0.22.1, 0.22, 0.21.3, 0.20.3 and 0.19.2.


user: DSOTM-pf
labels: Needs Investigation: Issue requires investigation
issue_23411: state: open
title: HalvingRandomSearchCV - Custom Factor
body: ### Describe the workflow you want to enable

There is often the problem with successive Halving that there are too many candidates and not enough resources. At the moment you can handle this by using Aggressive Elimination or trying to adjust the halving factor.

I would like to enable a workflow where two factors are used, one for the number of candidates and one for the number of samples per round. Example: Increase data with factor of 2 each round, but dividde the candidates by 3.

### Describe your proposed solution

The solution would be rather easy:
 
1. Add a second factor_cand for candidates, which be default is not used
2. If the user sets factor_cand manually in gridsearch or by means of initialization it scales the number of candidates independently of the already implemented scaling factor.
3. Optionally: Add some flags and checks to see if this works out with the number of resources, though I see no issues here.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: nilslacroix
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23405: state: open
title: LassoLars: improve precision at lower regularization values
body: [LassoLars](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLars.html?highlight=lassolars#sklearn.linear_model.LassoLars) can be quite imprecise at low regularization values (e.g. alpha=alpha_max/1000).
This can be easily solved by modifying the definition of the variable `equality_tolerance` (on file linear_model/_least_angle.py) so that it becomes proportional to the value of alpha, for instance as follows:

`equality_tolerance = min(tiny32, np.finfo(np.float32).eps * alpha_min)`
user: cassiofragadantas
labels: Needs Investigation: Issue requires investigation
issue_23401: state: open
title: Grouping of infrequent categories in 𝗢𝗻𝗲𝗛𝗼𝘁𝗘𝗻𝗰𝗼𝗱𝗲𝗿
body: Dear all,

This feature is very similar to the one presented in this paper:
https://ieeexplore.ieee.org/document/8851888

which states the following paragraph:
The goal of the PCP transform is to substantially reduce the input memory and processing requirements while keeping the most relevant levels. Fig. 2 exemplifies the effect of this transform, by presenting the 1,000 highest frequency values (decreasing order) for the user city attribute for the TEST traffic data (which contains a total of 10,690 levels). For this attribute and when P = 10%, PCP selects only the most frequent 688 levels (dashed vertical line in Fig. 2), merging the other 10,002 infrequent levels into the "Others" label. 

<img width="883" alt="Captura de ecrã 2022-05-17, às 16 13 10" src="https://user-images.githubusercontent.com/39952471/168860640-dbe54b09-2771-4c9f-a0d9-17d4058bf29b.png">

Just asking, did the authors took notice of this paper? It can also improve this feature (instead of using a fixed number of categories it could be using a percentage categorical values to keep). 


Thanks for the attention!
user: Metalkiler
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23386: state: open
title: FET support feature selection based on permutation importance
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Attempt to address #15075

#### What does this implement/fix? Explain your changes.
This PR adds another parameter `importance_type` to `feature_selection.SelectFromModel`, which could be set to "permutation" to use permutation importance for feature selection.

Since permutation importance can be calculated for any other validation sets, user will need to call `fit` method to calculate the importance score before calling `transform`.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: MaxwellLZH
labels: Needs Decision: Requires decision
issue_23382: state: open
title: CV integration for OOB-scoring
body: ### Describe the workflow you want to enable

Out-of-Bag (OOB) scoring provides an estimate of the model generalizability for `RandomForest` without needing to refit the model several times as is demanded by k-fold cross validation (CV). Although `sklearn` provides a mechanism to obtain this estimate, it does *not* provide a mechanism to integrate it into the existing cross validation workflows. For example, we might have a `GridSearchCV` where we want to optimise hyperparameters for the forest, but only fitting *once* per parameter set. This in theory could be implemented using OOB error. 

As far as I can, the two parameters of interest here are `cv` and `scoring`, both inputs to all the CV-related classes, and ultimately to `cross_val_score()`. `scoring` can be implemented easily enough using a [custom scorer](https://scikit-learn.org/stable/modules/model_evaluation.html#implementing-your-own-scoring-object), since this has access to the final estimator and therefore the OOB error. What is problematic here is the `cv` argument, which *requires* that we split the dataset, and offers no alternative.

### Describe your proposed solution

* We add `sklearn.metrics.oob`, a scoring function that just returns the oob error on the trained classifier
* We add `sklearn.model_selection.IntegratedCV`, which is a cross validator that **does not split the data at all**. ie `IntegratedCV().split(X)` will return `X` unchanged

With the combination of these two entities, users will be able to perform OOB-based cross-validation

### Describe alternatives you've considered, if relevant

It is possible to apply general cross validation metrics to a `RandomForest`, such as k-folds. This is an alternative that already exists in sklearn today. However we are neglecting the significant (`k` times) speedup that could be obtained using OOB error.

### Additional context

This question is notably discussed in these threads:
* https://datascience.stackexchange.com/a/66238/83633
* https://datascience.stackexchange.com/questions/37393/scikitlearn-grid-search-random-forest-using-oob-as-metric
* https://datascience.stackexchange.com/questions/76304/gridsearchcv-with-random-forest-classifier
user: multimeric
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23376: state: open
title: The data type of input data for LinearRegression class will affect the results
body: ### Describe the bug

Our team just used the class sklearn.linear_model.LinearRegression to do multi-linear regression. And, we found out that the same data, which means the values are identical for each element, with different data formats will return different coefficients. The array we use are all integer values but saved as float32 and float64, separately. However, the array in float64 as input get us the expected result, but float32 doesn't.

The code below is a simple reproducible example.

### Steps/Code to Reproduce

```python
In [1]: from sklearn.linear_model import LinearRegression
    ...: import numpy as np
    ...: rng = np.random.default_rng(0)
    ...: x_1_array = rng.integers(low=1000, high=10000, size=10_000_000)
    ...: x_2_array = rng.integers(low=1000, high=10000, size=10_000_000)
    ...: x_3_array = rng.integers(low=1000, high=10000, size=10_000_000)
    ...: y_array = 3 * x_1_array + x_2_array - 5 * x_3_array
    ...: intercept_data = rng.integers(low=-3, high=3)
    ...: print(f'True intercept {intercept_data}')
    ...: y_array += intercept_data
    ...: X_train = np.stack([x_1_array,x_2_array,x_3_array],axis=-1)
    ...: y_train = y_array
    ...: linreg = LinearRegression()
    ...: model = linreg.fit(X_train.astype(np.float64), y_train)
    ...: print(f'Float 64 Intercept: {linreg.intercept_}')
    ...: print(f'Float 64 Coefficient: {linreg.coef_}')
    ...: model = linreg.fit(X_train.astype(np.float32), y_train)
    ...: print(f'Float 32 Intercept: {linreg.intercept_}')
    ...: print(f'Float 32 Coefficient: {linreg.coef_}')
```

### Expected Results

![image](https://user-images.githubusercontent.com/51532833/168492124-d98f9663-4899-47ee-83b2-4952462c37a6.png)

### Actual Results

![image](https://user-images.githubusercontent.com/51532833/168492136-4e0f4100-9e48-4f58-aa2a-97e826f06c61.png)


### Versions

```shell
System:
    python: 3.7.9 (v3.7.9:13c94747c7, Aug 15 2020, 01:31:08)  [Clang 6.0 (clang-600.0.57)]
executable: /Users/lazy/PycharmProjects/projects/bin/python
   machine: Darwin-21.4.0-x86_64-i386-64bit

Python dependencies:
          pip: 21.3
   setuptools: 58.2.0
      sklearn: 1.0.2
        numpy: 1.21.4
        scipy: 1.7.2
       Cython: None
       pandas: 1.3.4
   matplotlib: 3.4.3
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

user: zhenliu26
labels: float32: Issues related to support for 32bit data
issue_23371: state: open
title: FIX apply sample weight to RANSAC residual threshold
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Addresses #15836.

#### What does this implement/fix? Explain your changes.
There is another opening PR #15952 that applies sample weight when calculating `residuals_subset`, meanwhile in this PR, sample weight is used to adjust `residual_threshold`. 

The reason for this change follows @glemaitre 's comment in the original PR https://github.com/scikit-learn/scikit-learn/pull/15952#issuecomment-568606084.

For example, if we set `sample_weight` to large values for samples that we want to be included in the model, in the original PR, this will result in a large residual and those samples will be considered outliers. In this PR, the `residual_threshold` is calculated using weighted sum, so those samples with large weights will more likely to be included in the final model, which I think follows intuition.

#### Other comments
I'm not 100% sure adjusting `residual_threshold` is the correct way for using sample_weight, so it's open for discussion
<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: MaxwellLZH
issue_23361: state: open
title: Enable loading parquet file from OpenML
body: After a discussion [here](https://github.com/openml/OpenML/issues/1133), we can rely on `parquet` to load data from OpenML.

Since we introduce the `parser` keyword, we can provide `parser="parquet"` that can rely on some optional dependencies to load the dataset.

In addition, we should check the consistency between `parser="pandas"` and `parser="parquet"`.
user: glemaitre
issue_23346: state: open
title: FIX Enable SelfTrainingClassifier to work with vectorizers
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes https://github.com/scikit-learn/scikit-learn/issues/23323


#### What does this implement/fix? Explain your changes.
This PR tries to delegate most responsibilities to the base estimator in `SelfTrainingClassifier`. Note this PR will fail this test:

https://github.com/scikit-learn/scikit-learn/blob/a47d569e670fd4102af37c3165c9b1ddf6fd3005/sklearn/utils/estimator_checks.py#L3076

because it does not cast the "NotAnArray" to a ndarray, which means it can not index it anymore with `_safe_indexing`. The only work around I see is for `_safe_indexing` to call `np.asarray` so the "NotAnArray" can be indexed

#### Any other comments?
I have an PR with a similar issue at https://github.com/scikit-learn/scikit-learn/pull/21811. In that case `Bagging` is trying to index the array in both axis.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
issue_23334: state: open
title: API to predict multiple quantiles at once
body: Classifiers have a `predict_proba` method that makes it possible to quantify probabilistic ally the certainty in the predictions for a given input `X_i`.

Currently most regressors in scikit-learn only predict a conditional expectile E[Y|X], and some have a `return_std` option that makes it also possible to estimate sqrt(VAR[Y|X]), which can be used to quantify the certainty when assuming a Gaussian predictive distribution (typically for Gaussian processes which estimate a Gaussian predictive posterior distribution).

We do have pointwise quantile estimators (linear models, gradient boosting, hist gradient boosting) where the `predict` method returns a single point estimate for target quantile passed as an hyper-parameter instead of estimating an expectile.

Several people have expressed the need to have more generic API that can return an array of quantile estimates for a given input `X_i`.

The goal of this issue is to centralize the discussion of an API extension to be able to do this more uniformly in scikit-learn, either via a meta-estimator that wraps an array of point-wise quantile estimator to turn it into a quantile-array estimator or to directly have the base estimators able to do this directly (and sometimes more efficiently).

Some non-exhausitive list of related PRs and issues (feel free to add or suggest new ones):

- https://github.com/scikit-learn/scikit-learn/issues/19851
- https://github.com/scikit-learn/scikit-learn/pull/19754
- https://github.com/scikit-learn/scikit-learn/issues/20964

Also related:

- conformal predictions: https://github.com/scikit-learn-contrib/MAPIE
- XGBoostLSS (location, scale and shape) https://github.com/StatMixedML/XGBoostLSS

Furthermore, models like Poisson regression that make a specific assumption about the conditional Y|X distribution, it would be possible to estimates of the inverse-CDF values of the estimated Y|X for instance. Those could probably also benefit from an expanded API.

If we do this, then we have the side question of how to evaluate such multi-quantile models. We could probably extend the pinball_loss scorer to average the pinball scores for an array of quantiles for instance.

/cc @GaelVaroquaux @amueller @lorentzenchr 
user: ogrisel
labels: Needs Decision - API: Requires decision regarding API
issue_23324: state: open
title: Support for ordinal multi-classification
body: ### Describe the workflow you want to enable

Encode response variable ordering with every scikit learn classifier according to the method introduced in this [frequently cited paper](https://link.springer.com/chapter/10.1007/3-540-44795-4_13).

### Describe your proposed solution

Implement `OrdinalClassifier`, a classifier that takes another scikit learn classifier as input and encodes the ordinality assumption into the classifier. It's quite simple, and there are also blog posts on it, for example by @M46F [here](https://towardsdatascience.com/simple-trick-to-train-an-ordinal-regression-with-any-classifier-6911183d2a3c). Here's a slightly more advanced version that can also handle arbitrary labels that I'd be willing to contribute as a PR: https://gist.github.com/lorenzwalthert/51371894225c7b530b66bdabfad60327

A lot of classification problems are inherently ordinal in nature and this trick has proven to work well in various settings, the code is very little to add to the project, adds zero dependencies and is a much cited approach (600+).

### Describe alternatives you've considered, if relevant

Using that code outside of scikit learn, but I think ti's generally useful to others.

### Additional context

_No response_
user: lorenzwalthert
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23323: state: open
title: SelfTrainingClassifier on a Pipeline
body: ### Describe the bug

SelfTrainingClassifier cannot be fit on text data even if the base_estimator parameter is an estimator that can accept text data (e.g. a pipeline with text preprocessing). In particular, it seems that SelfTrainingClassifier validates the data on the classifier within the pipeline (hence being unable to accept text) instead of validating on the entire pipeline (which can accept text).

### Steps/Code to Reproduce

```python
import pandas as pd
from sklearn.datasets import fetch_20newsgroups
from sklearn.compose import ColumnTransformer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.semi_supervised import SelfTrainingClassifier

#pulling text data
categories = ['alt.atheism', 'soc.religion.christian']
twenty_train = fetch_20newsgroups(subset='train', categories=categories, shuffle=True, random_state=42)

X = pd.DataFrame({'text': twenty_train.data})
y = twenty_train.target
#changing some labels to -1 for semi supervised
y[len(y) - 100:len(y)] = -1

#building pipeline
preprocessor = ColumnTransformer(
    transformers =  [
        ('vectorizer', CountVectorizer(), 'text')
    ])

classifier = MultinomialNB()

pipe = Pipeline(steps = [('preprocessor', preprocessor), ('classifier', classifier)])

#fitting SelfTrainingClassifier with the pipeline as the base estimator
stc = SelfTrainingClassifier(pipe)
stc.fit(X,y)

#note that fitting pipe does work
#pipe.fit(X,y)
```

### Expected Results

No error is thrown.

### Actual Results

```python-traceback
ValueError                                Traceback (most recent call last)
Input In [27], in <cell line: 30>()
     28 #fitting SelfTrainingClassifier with the pipeline as the base estimator
     29 stc = SelfTrainingClassifier(pipe)
---> 30 stc.fit(X,y)

File ~/code/scikit-learn/sklearn/semi_supervised/_self_training.py:181, in SelfTrainingClassifier.fit(self, X, y)
    162 """
    163 Fit self-training classifier using `X`, `y` as training data.
    164 
   (...)
    177     Fitted estimator.
    178 """
    179 # we need row slicing support for sparce matrices, but costly finiteness check
    180 # can be delegated to the base estimator.
--> 181 X, y = self._validate_data(
    182     X, y, accept_sparse=["csr", "csc", "lil", "dok"], force_all_finite=False
    183 )
    185 if self.base_estimator is None:
    186     raise ValueError("base_estimator cannot be None!")

File ~/code/scikit-learn/sklearn/base.py:596, in BaseEstimator._validate_data(self, X, y, reset, validate_separately, **check_params)
    594         y = check_array(y, input_name="y", **check_y_params)
    595     else:
--> 596         X, y = check_X_y(X, y, **check_params)
    597     out = X, y
    599 if not no_val_X and check_params.get("ensure_2d", True):

File ~/code/scikit-learn/sklearn/utils/validation.py:1074, in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
   1069         estimator_name = _check_estimator_name(estimator)
   1070     raise ValueError(
   1071         f"{estimator_name} requires y to be passed, but the target y is None"
   1072     )
-> 1074 X = check_array(
   1075     X,
   1076     accept_sparse=accept_sparse,
   1077     accept_large_sparse=accept_large_sparse,
   1078     dtype=dtype,
   1079     order=order,
   1080     copy=copy,
   1081     force_all_finite=force_all_finite,
   1082     ensure_2d=ensure_2d,
   1083     allow_nd=allow_nd,
   1084     ensure_min_samples=ensure_min_samples,
   1085     ensure_min_features=ensure_min_features,
   1086     estimator=estimator,
   1087     input_name="X",
   1088 )
   1090 y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
   1092 check_consistent_length(X, y)

File ~/code/scikit-learn/sklearn/utils/validation.py:856, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
    854         array = array.astype(dtype, casting="unsafe", copy=False)
    855     else:
--> 856         array = np.asarray(array, order=order, dtype=dtype)
    857 except ComplexWarning as complex_warning:
    858     raise ValueError(
    859         "Complex data not supported\n{}\n".format(array)
    860     ) from complex_warning

File ~/mambaforge/envs/dev/lib/python3.10/site-packages/pandas/core/generic.py:2064, in NDFrame.__array__(self, dtype)
   2063 def __array__(self, dtype: npt.DTypeLike | None = None) -> np.ndarray:
-> 2064     return np.asarray(self._values, dtype=dtype)

ValueError: could not convert string to float: 'From: [...]'
```

### Versions

```shell
System:
    python: 3.10.4 

Python dependencies:
          pip: 22.0.4
   setuptools: 62.1.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.7.3
       Cython: None
       pandas: 1.4.2
   matplotlib: None
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

user: AshwinPuri13
issue_23319: state: open
title: Yeo-Johnson Power Transformer gives Numpy warning
body: ### Describe the bug

When I use a power transformer with yeo-johnson method I get this warning in numpy:

`../lib/python3.10/site-packages/numpy/core/_methods.py:235: RuntimeWarning: overflow encountered in multiply`

Strangely I can't surpress this warning with filter.warnings()

**Tl;dr: The eroor seems to appear when multiple instances of the transformer are called, in this example because n_jobs=2 is set in the preprocessor. But the bug also appears with n_jobs=1 when you call this preprocessor in a grid search for example.**

### Steps/Code to Reproduce

The following code snippet will raise a warning:

```python
import numpy as np
from sklearn.preprocessing import PowerTransformer

rng = np.random.default_rng(0)
X = rng.exponential(size=(50,))
X -= X.max()
X = -X
X += 100
X = X.reshape(-1, 1)

method = "yeo-johnson"
transformer = PowerTransformer(method=method, standardize=False)
transformer.fit_transform(X)
```

### Expected Results

No Warnings

### Actual Results
```pytb
/Users/glemaitre/mambaforge/envs/dev/lib/python3.8/site-packages/numpy/core/_methods.py:233: RuntimeWarning: overflow encountered in multiply
  x = um.multiply(x, x, out=x)

```

### Versions

```shell
System:
    python: 3.8.12 | packaged by conda-forge | (default, Sep 16 2021, 01:38:21)  [Clang 11.1.0 ]
executable: /Users/glemaitre/mambaforge/envs/dev/bin/python
   machine: macOS-12.3.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.2.dev0
          pip: 21.3
   setuptools: 58.2.0
        numpy: 1.21.6
        scipy: 1.8.0
       Cython: 0.29.24
       pandas: 1.4.2
   matplotlib: 3.4.3
       joblib: 1.0.1
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /Users/glemaitre/mambaforge/envs/dev/lib/libopenblas_vortexp-r0.3.18.dylib
        version: 0.3.18
threading_layer: openmp
   architecture: VORTEX
    num_threads: 8

       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /Users/glemaitre/mambaforge/envs/dev/lib/libomp.dylib
        version: None
    num_threads: 8
```

user: nilslacroix
issue_23288: state: open
title: Less greedy RFECV (1-S.E. Rule)
body: Current implementation of RFECV selects the number of features that achieve optimal score (e.g., accuracy, r2).
This can lead to a greedy behavior and i feel it would be useful to have a "One-standard-error" rule for selecting the number of features. This is feasible now that the `cv_results_` is returned instead of `grid_scores_`

For example. Looking at the [User Guide Example](https://scikit-learn.org/stable/auto_examples/feature_selection/plot_rfe_with_cross_validation.html#sphx-glr-auto-examples-feature-selection-plot-rfe-with-cross-validation-py), RFECV selects 6 features (maximum score) although it is easy to see that 3 features achieve a score within 1S.E. of the maximum score using error bars. These 3 features also coincide with the 3 informative features used in the simulation 


![class_rfecv](https://user-images.githubusercontent.com/25321020/166947385-1b801ccd-0430-476c-8bfc-147cc134a5b4.png)

Same goes for the Boston Dataset. RFECV selects 6 features (minimum MSE) but 4 features achieve performance within 1S.E.
![boston_rfecv](https://user-images.githubusercontent.com/25321020/166947665-edf59272-57db-4c87-a5e1-d27ca22f12f9.png)

In a real project with hundreds of features this approach could lead to a significant reduction of features with minimal effect on performance. 

user: DSoudis
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23286: state: open
title: FET add support for manhattan distances in KNN imputer
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### What does this implement/fix? Explain your changes.
Manhattan distances is quite useful when data have outliers, this PR adds a new pairwise metric `nan_manhattan_distances` so that it can be used in `KNNImputer`.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: MaxwellLZH
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23243: state: open
title: Improve SVR vs KRR example further
body: Follow up of https://github.com/scikit-learn/scikit-learn/pull/22804, example code is https://github.com/scikit-learn/scikit-learn/blob/main/examples/miscellaneous/plot_kernel_ridge_regression.py

- once the grid-search is done use `svr.best_estimator_` and `kr.best_estimator_` as mentioned in https://github.com/scikit-learn/scikit-learn/pull/22804#discussion_r825469629. Maybe using the same gamma and an "equivalent" `C = 1 / alpha` was done on purpose to be able to compare both models in a more fair fashion, not entirely sure.
- Add train scores to learning curve and maybe use 1000 samples rather than 100 samples: https://github.com/scikit-learn/scikit-learn/pull/22804#issuecomment-1066134467. When doing that the learning curve don't make too much sense, try to understand why ...
- timings can change between two runs, not sure what to do, doing more runs will make the example slower and the example is already taking ~20s. Also this may be sensitive to `OPENBLAS_NUM_THREADS` (or `MKL_NUM_THREADS`). 
user: lesteve
issue_23233: state: open
title: Move scikit-learn.org/dev into a seperate repo
body: Currently, https://github.com/scikit-learn/scikit-learn.github.io grows quickly because of the pushes from `main` and the binaries in the diff (images and the zipped HTML). When I clone the repo locally, the folder is 8.8 GB.

I think it's better to follow what matplotlib does: https://github.com/matplotlib/devdocs, where we force push `main` onto a "devdocs" repo, and then have `scikit-learn.com/dev` point to that repo. In release branches, we can still push to https://github.com/scikit-learn/scikit-learn.github.io.

Related to https://github.com/scikit-learn/scikit-learn/issues/21171
user: thomasjpfan
issue_23224: state: open
title: FIX "CategoricalNB fails with sparse matrix" #16561
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes #16561.
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
I fixed a bug where even though the documentation says `CategoricalNB` accepts `sparse_matrix` as a feature matrix, it cannot.

Now `CategoricalNB` accepts a sparce matrix (in particular, csr_matrix) without any error.
Also, I added the test for it.


#### Any other comments?

In #16561, one of the maintainers asked in what situation you want CategoricalNB for sparse data.

When I did an assignment in my university class, I was required to train a very simple model like naive Bayes algorithms with sparse data, which is originally text data.
If `toarray()` was used, the memory error occurred. 

I created such an example. If I run the following code on my local computer with 16GB memory, the process was killed becaues of the memory shortage if I didn't use a sparse matrix. If I did, I was able to successfully finish it.

```
from sklearn.naive_bayes import CategoricalNB, MultinomialNB
from scipy.sparse import csr_matrix
import numpy as np

sample_size = 1000
n_features = 50000
n_ones = 100
is_sparse = True

rng = np.random.RandomState(1)
rows = rng.choice(sample_size, size=n_ones)
cols = rng.choice(n_features, size=n_ones)

if is_sparse:
    X = csr_matrix((np.ones(n_ones), [rows, cols]), shape=(sample_size, n_features))
else:
    X = np.zeros((sample_size, n_features))
    X[rows, cols] = 1
y = rng.choice(2, size=sample_size)
clf = CategoricalNB(min_categories=3, class_prior=np.array([0.3, 0.7]))
clf.fit(X, y)
```

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: g-morishita
issue_23211: state: open
title: Grouped timings for parametrized tests
body: Sometimes it is convenient to group parametrized tests by function name, to look at the total time taken by the group of parametrized tests, rather than individual parametrized tests. When trying multiple combinations of parameters, it is easy to generate a big number of parametrized tests without realizing the total test time.

cc @jeremiedbb, something like this could be added CI if we think this is worth it. Right now we are only showing the 20 slowest individual tests we could add the 20 slowests tests with grouped parametrized tests.

To write the tests timings info to a file, add this to sklearn/conftests.py:
```py
def pytest_terminal_summary(terminalreporter, exitstatus, config):
    # use @ as CSV delimiter , ; and | are arising in test names so can not be used ...
    lines = ['name@duration\n']
    for reps in terminalreporter.stats.values():
        for rep in reps:
            if getattr(rep, 'when', None) == "call":
                line = f"{rep.nodeid}@{rep.duration}\n"
                lines.append(line)

    with open('/tmp/test_timings.csv', 'w') as test_timings_file:
        test_timings_file.writelines(lines)
```

To analyse the CSV:
```py
import pandas as pd

pd.options.display.max_colwidth = None
df = pd.read_csv('/tmp/test_timings.csv', delimiter='@')
df['function_name'] = df['name'].str.extract(r'(.+?)\[.+\]')
df['function_name'] = df['function_name'].where(df['function_name'].notna(), df['name'], axis=0)
print(df.groupby('function_name')['duration'].aggregate(['sum', 'size']).sort_values(by='sum', ascending=False).iloc[:50])
```

On my machine:
```                                                                                                                   
                                                                                                                 sum        size
function_name                                                                                                                   
sklearn/tests/test_common.py::test_estimators                                                                    90.395519  7291
sklearn/cluster/tests/test_k_means.py::test_k_means_fit_predict                                                  25.514541    32
sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch                                         23.514811   108
sklearn/tests/test_common.py::test_search_cv                                                                     11.408637   640
sklearn/tree/tests/test_tree.py::test_prune_tree_regression_are_subtrees                                         10.413731    88
sklearn/inspection/_plot/tests/test_plot_partial_dependence.py::test_plot_partial_dependence_kind                 9.233481    24
sklearn/inspection/_plot/tests/test_plot_partial_dependence.py::test_partial_dependence_plot_limits_one_way       7.611308    12
sklearn/inspection/tests/test_partial_dependence.py::test_output_shape                                            7.524255   144
sklearn/tests/test_common.py::test_pandas_column_name_consistency                                                 7.299764   229
sklearn/decomposition/tests/test_dict_learning.py::test_minibatch_dictionary_learning_positivity                  6.513909    12
sklearn/linear_model/tests/test_quantile.py::test_quantile_estimates_calibration                                  6.451284     3
sklearn/model_selection/tests/test_split.py::test_stratified_kfold_ratios                                         6.122864    28
sklearn/tests/test_docstrings.py::test_docstring                                                                  6.090240  1803
sklearn/inspection/_plot/tests/test_plot_partial_dependence.py::test_plot_partial_dependence_lines_kw             5.780402    12
sklearn/tree/tests/test_tree.py::test_prune_tree_classifier_are_subtrees                                          5.337500    36
sklearn/decomposition/_dict_learning.py::sklearn.decomposition._dict_learning.DictionaryLearning                  5.231721     1
sklearn/utils/tests/test_estimator_checks.py::test_check_estimator_clones                                         4.757799     1
sklearn/tests/test_common.py::test_check_n_features_in_after_fitting                                              4.365753   195
sklearn/tests/test_calibration.py::test_calibration_multiclass                                                    4.298814     8
sklearn/ensemble/tests/test_forest.py::test_memory_layout                                                         4.225799     8
sklearn/_loss/tests/test_loss.py::test_loss_dtype                                                                 3.981264  1024
sklearn/neighbors/tests/test_neighbors.py::test_knn_forcing_backend                                               3.858353    20
sklearn/inspection/_plot/tests/test_plot_partial_dependence.py::test_partial_dependence_overwrite_labels          3.802329    12
sklearn/cluster/tests/test_k_means.py::test_predict                                                               3.793582    24
sklearn/tests/test_docstring_parameters.py::test_fit_docstring_attributes                                         3.763307   203
sklearn/inspection/tests/test_partial_dependence.py::test_partial_dependence_dataframe                            3.611114    12
sklearn/linear_model/tests/test_quantile.py::test_sparse_input                                                    3.534900    18
sklearn/mixture/tests/test_gaussian_mixture.py::test_init                                                         3.497789     1
sklearn/inspection/_plot/tests/test_plot_partial_dependence.py::test_plot_partial_dependence_passing_numpy_axes   3.449946     6
sklearn/ensemble/tests/test_forest.py::test_sparse_input                                                          3.325918    15
sklearn/linear_model/tests/test_sgd.py::test_SGDClassifier_fit_for_all_backends                                   3.193899     3
sklearn/metrics/tests/test_pairwise_distances_reduction.py::test_strategies_consistency                           3.174157    56
sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_sample_weight_effect               3.005303     6
sklearn/ensemble/tests/test_forest.py::test_importances                                                           2.923119    20
sklearn/linear_model/tests/test_ridge.py::test_ridge_gcv_vs_ridge_loo_cv                                          2.905850    48
sklearn/cluster/tests/test_optics.py::test_dbscan_optics_parity                                                   2.799847     9
sklearn/inspection/_plot/tests/test_plot_partial_dependence.py::test_plot_partial_dependence_str_features         2.790852    26
sklearn/manifold/tests/test_spectral_embedding.py::test_spectral_embedding_precomputed_affinity                   2.748455     8
sklearn/model_selection/tests/test_successive_halving.py::test_aggressive_elimination                             2.747456     8
sklearn/neural_network/tests/test_mlp.py::test_n_iter_no_change_inf                                               2.743698     1
sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_missing_values_resilience          2.473407     6
sklearn/impute/tests/test_impute.py::test_iterative_imputer_no_missing                                            2.403574     1
sklearn/ensemble/tests/test_weight_boosting.py::test_adaboost_consistent_predict                                  2.401502     2
sklearn/linear_model/tests/test_ridge.py::test_ridge_gcv_sample_weights                                           2.219063    32
sklearn/decomposition/tests/test_dict_learning.py::test_warning_default_transform_alpha                           2.208973     2
sklearn/linear_model/tests/test_quantile.py::test_equivariance                                                    2.128632     3
sklearn/neighbors/tests/test_neighbors.py::test_neigh_predictions_algorithm_agnosticity                           1.918901   168
sklearn/linear_model/tests/test_quantile.py::test_quantile_sample_weight                                          1.890437     1
sklearn/metrics/tests/test_pairwise_distances_reduction.py::test_pairwise_distances_radius_neighbors              1.859356    56
sklearn/experimental/tests/test_enable_iterative_imputer.py::test_imports_strategies                              1.771305     1
```

user: lesteve
issue_23205: state: open
title: Exposing submodules directly from main sklearn module instead of recursively importing each submodule separately
body: ### Describe the workflow you want to enable

The problem that I'm facing is the same as the issue described in this [stackoverflow](https://stackoverflow.com/questions/46572475/module-sklearn-has-no-attribute-cross-validation#comment127202377_46572475) page. Basically sklearn lacks the ability to allow users flexibility in how they would like to import submodules.

For instance, if someone wants to use `x,y,z,...,etc` submodule form sklearn they would have to do the following for each of them:
```
from sklearn.x import x
from sklearn.y import y
from sklearn.z import z
```

This not only increases the line of codes in a codebase but also makes core readability more difficult to follow, especially in large code bases. 

### Describe your proposed solution

The best thing would be to enable the same functionality as numpy.
For instance, in numpy all we have to do is `import numpy as np` and all the magic is in the `np.` namespace.
I would really like to be able to do the same thing with sklearn, e.g., `import sklearn as sk` and then calling the different submodules as needed, e.g., `sk.metrics.auccuracy` or `sk.model_selection.cross_validate`, etc.

At least this is a coherent way of reading the code and understanding much easier where each submodule came from.

I really don't know why sklearn has avoided this kind of exposure of submodules since all other major libraries do it, e.g., `import pandas as pd`?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: kirk86
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23199: state: open
title: Consider LSMR instead of LSQR
body: #### Proposition
Currently, we use the LSQR solver for (possibly sparse) linear equation / least squares problems in
- `LinearDiscriminantAnalysis`
- `LinearRegression`
- `Ridge`

Consider to change to LSMR because according to [1] and [2] LSMR might be able to terminate earlier and be faster. See in particular Fig 1 of [2] which compares LSMR and LSQR on CUTEst and UFL test sets of 921 eligible problems with no preconditioning.

If we decide to do so, how do we implement the change?
- Just replace `lsqr` with `lsmr` and document it; or
- Add new solver option `lsmr`.


#### References
[1] https://stanford.edu/group/SOL/software/lsmr/

[2] Gould, N.I., & Scott, J.A. (2017). The State-of-the-Art of Preconditioners for Sparse Linear Least-Squares Problems. ACM Transactions on Mathematical Software (TOMS), 43, 1 - 35.
   http://centaur.reading.ac.uk/70342/1/ls_study_toms.pdf
user: lorentzenchr
issue_23195: state: open
title: Add more robust Kmeans initializer for mixture.GaussianMixture
body: ### Describe the workflow you want to enable

The [GaussianMixture](https://github.com/scikit-learn/scikit-learn/blob/24f3006fb2d054b8afb26382209ae33629a8dfe0/sklearn/mixture/_gaussian_mixture.py#L456 ) class has an option of initializing using `K-means`, another method, or passing in your own initializer values, with the default being `K-Means`. 

The expectation maximization (EM) algorithm used by the `.fit` method of this class is very sensitive to initial conditions, as also mentioned in the docs. However, the `K-means` initializer uses hardcoded parameters which are passed via a super() call to the parent [BaseMixture](https://github.com/scikit-learn/scikit-learn/blob/24f3006fb2d054b8afb26382209ae33629a8dfe0/sklearn/mixture/_base.py#L50) class.

The workflow I want to enable is one where a more robust K-means initializer can be readily used (~1 line code) to initialize the GMM model. 

### Describe your proposed solution

There are two simple possibilities, which can be implemented alongside each other:

1. Allow passing in a `initializer_kwargs = {}` or `kmeans_kwargs` dictionary to the initializer, which is empty/None by default. These kwargs should be unpacked and overwrite the  keyword arguments passed to the initializer algorithm. Allowing optionally using a more robust but costly set of parameters.    
2. Provide a helper function in `mixtures` or a method to `cluster.Kmeans` to produce the desired inputs expected by `mixture.GaussianMixture`. This logic is currently buried in a mix of the [base class](https://github.com/scikit-learn/scikit-learn/blob/24f3006fb2d054b8afb26382209ae33629a8dfe0/sklearn/mixture/_base.py#L128) and `GaussianMixture` class and uses private ('_'functions) which are not exposed in `sklearn.mixture`, so need to be absolute imported. 

The benefit to option 1 is that the docstring of `GaussianMixture` can be modified to easily explain this and provide an example. It also would take the least amount of code. For example `sklearn.mixture.GaussianMixture(kmeans_kwargs={'n_init':1}). It is also immune to breaking changes in either cluster.Kmeans or mixture.GaussianMixture. 

Option 2 has the nice property that it can be used internally inside of mixture.GaussianMixture, instead of the adhoc current code. This can be especially useful if this option is propagated to other GMM mixture classes such as the Bayesian one. It also allows one mess around until an optimal initializer is found by experimenting with the KMeans class, and finally taking the desired result to use to init the GMM class.

### Describe alternatives you've considered, if relevant

It is already easy, especially if one understand the codebase or the math, to run your own Kmeans (or other method) as an initializer. Take the output, calculate the required parameters, which is actually the responsibilities, but  `means` (provided), `weights` (not-provided by Kmeans) and technically `precisions` (inverse of covariance matrix, although this can be calculated inside the class). This however takes several lines of code and benefits from using private/non-exposed functions inside mixture module. Therefore, it's not an ideal alternative.

Another alternative is to use composition and allow passing in an instance of a clustering algorithm class (like Kmeans), with the parameters already set. While this is useful, in practice it would mean validating many other clustering algorithms in sklearn to make sure they are usable, or requiring users to write their own decorators (or class extensions) to provide the values needed by GMM. I think it's too technically onerous for the small gain, even though I would love more composition in sklearn between classes and modules. This would also mean adding a potentially confusing extra way to pass an init_params argument that is a callable instead of a string, or modifying that parameter to confusingly accept strings as well as callables (instances of a clustering or other initializer algorithm). 

Final alternative is to allow passing in user-derived `responsibilities` instead of the above 3 separate parameters, which are actually what the super initialize class uses and all that is needed; As well as, provide a property on Kmeans which calculates the responsibilities. This has the added benefit that users can easily get responsibilities from Kmeans and that users only have to pass a single parameter instead of 3. But it would mean a bit more involved (still relatively easy) rewrite of GaussianMixtures class. However, it would also then likely deviate from other mixture classes and introduce an change to Kmeans.

**TL;DR**: Add: (1) optional keyword arguments to GMM class that is passed to the initializer. (2) public helper function to convert user-derived K-means model to easily initialize a GMM class with 1 line of code. Alternatives (such as using composition) are valid, even preferable, but likely too much of a code review and maintenance burden for the minor improvement. The two proposed solutions add minimal review and maintenance burdens. 

### Additional context

In version 1.1 of sklearn `kmeans++` will be added as an alternative initializer, however, while better, it also suffers from the lack of being able to change its parameters. Additionally, it's not a very clear name so Kmeans will likely still be often used with a non-robust set of parameters. 

**I would like to submit a PR on this** if that's acceptable to the maintainers. But since it would be my first PR to sklearn, I would appreciate any feedback on my proposal so I can get a better understanding of how contributing works for this package.
user: emirkmo
issue_23183: state: open
title: ENH add zero_division=nan for classification metrics
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Fixes https://github.com/scikit-learn/scikit-learn/issues/22625


#### What does this implement/fix? Explain your changes.

This is an extension of https://github.com/scikit-learn/scikit-learn/pull/14900, where I added the parameter `zero_division` for precision, recall, and f1. Afterwards, it was added for jaccard as well.

Here, we add the ability to set zero_division to np.nan, so that np.nan is returned when the metric is undefined. In addition to this:
* when there is an average the numbers that are np.nan (due to undefined and then zero_division) are excluded from the average.
* when beta=0, return precision
* when just one of (precision, recall) is defined and it's 0, return fscore=0. Even if the other metric is undefined.

Specifically:

Precision:

* If pred_sum = 0, undefined
* If average != None, ignore from average any metric being np.nan

Recall:

If true_sum = 0, undefined
If average != None, ignore from average any class metric being np.nan

F-score:

if beta=inf, return recall, and beta=0, return precision
elif precision=0 or recall=0 (or both), return 0.   <------------- *this is a change*
else return zero_division
If average != None, ignore from average any metric being np.nan

Jaccard:

if all labels and pred are 0, return zero_division
If average != None, ignore from average any metric being np.nan


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: marctorsoc
issue_23180: state: open
title: Investigate SAG/SAGA solver
body: #### Description
The newly introduced tight tests for `Ridge` in #22910 together with the random seed fixture in #22749 revealed some shortcomings of the sag and saga solver:

1. ~~It shows some random behavior even with fixed random seed.~~
2. The `tol` needs to be set much smaller to receive comparable results with the other solvers of `Ridge`.

Ideally, the cause for both issues can be identified and fixed.

#### Some links for context
https://github.com/scikit-learn/scikit-learn/issues/23014
https://github.com/scikit-learn/scikit-learn/pull/23017
https://github.com/scikit-learn/scikit-learn/pull/23026
https://github.com/scikit-learn/scikit-learn/pull/23152
#23177


user: lorentzenchr
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_23179: state: open
title: Wrong normalization for pseudolikelihood in Restricted Boltzmann Machine.
body: ### Describe the bug

Pseudolikelihood of a sample under Restricted Boltzmann machine includes  the number of features  as a multiplier. 
Thus pseudolikelihood reflects the dimensionality of visible units, apart from the sample probability. 
I believe this multiplier should be removed. 

### Steps/Code to Reproduce

from sklearn.neural_network import BernoulliRBM
import numpy as np
 
RBM_= BernoulliRBM(verbose=1) 
X_10= np.ones([10,10])
X_1000= np.ones([10,1000])
                  

print('Learning RBM with 10 visible units:')
RBM_.fit(X_small)
print(100*'_')
print('Learning RBM with 1000 visible units')
RBM_.fit(X_large)
                  

### Expected Results

The pseudolikelihoods for 10 and 1000 visible units should be of the same order at corresponding learning stages.

### Actual Results

Learning RBM with 10 visible units:
[BernoulliRBM] Iteration 1, pseudo-likelihood = -0.19, time = 0.00s
[BernoulliRBM] Iteration 2, pseudo-likelihood = -0.28, time = 0.00s
[BernoulliRBM] Iteration 3, pseudo-likelihood = -0.22, time = 0.00s
[BernoulliRBM] Iteration 4, pseudo-likelihood = -0.12, time = 0.00s
[BernoulliRBM] Iteration 5, pseudo-likelihood = -0.10, time = 0.00s
[BernoulliRBM] Iteration 6, pseudo-likelihood = -0.07, time = 0.00s
[BernoulliRBM] Iteration 7, pseudo-likelihood = -0.07, time = 0.00s
[BernoulliRBM] Iteration 8, pseudo-likelihood = -0.08, time = 0.00s
[BernoulliRBM] Iteration 9, pseudo-likelihood = -0.10, time = 0.01s
[BernoulliRBM] Iteration 10, pseudo-likelihood = -0.06, time = 0.00s
____________________________________________________________________________________________________
Learning RBM with 1000 visible units
[BernoulliRBM] Iteration 1, pseudo-likelihood = -3.80, time = 0.01s
[BernoulliRBM] Iteration 2, pseudo-likelihood = -0.76, time = 0.02s
[BernoulliRBM] Iteration 3, pseudo-likelihood = -0.94, time = 0.01s
[BernoulliRBM] Iteration 4, pseudo-likelihood = -1.97, time = 0.01s
[BernoulliRBM] Iteration 5, pseudo-likelihood = -2.01, time = 0.01s
[BernoulliRBM] Iteration 6, pseudo-likelihood = -0.61, time = 0.01s
[BernoulliRBM] Iteration 7, pseudo-likelihood = -1.06, time = 0.01s
[BernoulliRBM] Iteration 8, pseudo-likelihood = -1.09, time = 0.01s
[BernoulliRBM] Iteration 9, pseudo-likelihood = -0.28, time = 0.01s
[BernoulliRBM] Iteration 10, pseudo-likelihood = -1.01, time = 0.01s


### Versions

```shell
System:
    python: 3.7.3 (default, Mar 27 2019, 22:11:17)  [GCC 7.3.0]
executable: /home/x/PycharmProjects/RBM/env/bin/python
   machine: Linux-4.15.0-142-generic-x86_64-with-debian-stretch-sid

Python dependencies:
          pip: 22.0.4
   setuptools: 40.8.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.7.3
       Cython: None
       pandas: None
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

user: KVasya
labels: Needs Investigation: Issue requires investigation
issue_23178: state: open
title: roc_auc_score must warn user about using labels instead of probabilities in binary case
body: ### Describe the workflow you want to enable

I've seen a lot of people use `YetAnotherClassifier.predict()` instead of desired `.predict_proba()[:, 1]` and get wrong result. This problem sometimes can be hard to found and IMO it can be very useful to explicitly warn user.

### Describe your proposed solution

Check if provided `y_score` has shape of `(n_samples, )` and that it contains 0 and 1. If so, it should simply warn user about a situation.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: overvalidated
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23172: state: open
title: Clarify and test dropping categories in linear models
body: #### Proposition
- [ ] Improve documentation about dropping or not dropping category levels.
- [ ] Add tests that linear models (with canonical link) satisfy for L2 penalized coefficients of one category: `sum_{levels} coef = 0`

#### Reasoning
We tell 2 different stories, e.g., in https://scikit-learn.org/dev/modules/generated/sklearn.preprocessing.OneHotEncoder.html and other places:

1. Use `OneHotEncoder` and do **not** drop any category level as that would break symmetry for penalized linear models. 
2. Use `drop="if_binary"`.

Those two points seem, at first glance, contradictory.

#### Background
https://doi.org/10.1093/biostatistics/kxm010 in Eq. (3.12) and following text, it becomes clear that summing over the L2 penalized coefficients of OHEded levels of one category (keeping all levels) yields zero.
Thus for binary features, keeping both levels would yield `coef_1 = - coef_2`. So dropping one level is reasonable (but results in a slightly different model if other features are present and if my math is correct).

user: lorentzenchr
labels: module:test-suite: everything related to our tests
issue_23162: state: open
title: MinCovDet estimation of covariance with strong bias?
body: ### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/23161

<div type='discussions-op-text'>

<sup>Originally posted by **hongfei0224** April 20, 2022</sup>
I was playing with MinCovDet (Minimum Covariance Determinant) in sklearn:
[https://scikit-learn.org/stable/modules/generated/sklearn.covariance.MinCovDet.html#sklearn.covariance.MinCovDet](url)

I see the idea of MinCovDet is to make the covariance matrix estimation robust when there is outlier in the data.
To understand the behavior, I tried the multivariate normal data **without** outliers.

What I did:
- generate multivariate_normal samples with given covariance matrix (the true covariance matrix)
- use `np.cov` to estimate empirical covariance matrix
- use `MinCovDet` to estimate robust covariance matrix
- repeat above with same "true covariance matrix"
- check the distribution of empirical covariance matrix and robust covariance matrix

I was expecting both `np.cov` and `MinCovDet` estimation should be centered around the true covariance matrix, since there is no outlier in data.

but seems not the case.

![image](https://user-images.githubusercontent.com/51205365/164061364-1daa7614-5178-46ad-b3be-0ff2e3c38a42.png)
![image](https://user-images.githubusercontent.com/51205365/164061425-00155926-f523-45a3-b872-ed83660aba5e.png)

the code to replicate results

```python
import pandas as pd
import numpy as np
from sklearn.covariance import EmpiricalCovariance, MinCovDet
import matplotlib.pyplot as plt
from tqdm.autonotebook import tqdm

res = []
res2= []
a = 2
b = 5
covar = 3 

for i in tqdm(range(1000)):
    x=np.random.multivariate_normal(mean=[0,0],cov=[[a,covar],[covar,b]], size=250).T
    res.append(np.cov(x))
    mcd = MinCovDet().fit(x.T)
    res2.append(mcd.covariance_)

pd.Series([xx[0][0] for xx in res]).hist(bins=30, label='empirical',alpha=0.7)
pd.Series([xx[0][0] for xx in res2]).hist(bins=30, label='cvd',alpha=0.7)
plt.axvline(a, ls='--', color='black')
plt.title('variance')
plt.legend()
plt.show()

pd.Series([xx[0][1] for xx in res]).hist(bins=30, label='empirical',alpha=0.7)
pd.Series([xx[0][1] for xx in res2]).hist(bins=30, label='cvd',alpha=0.7)
plt.axvline(covar, ls='--', color='black')
plt.title('covariance')
plt.legend()
plt.show()
```
user: hongfei0224
labels: Needs Investigation: Issue requires investigation
issue_23153: state: open
title: [FEAT] Implement quantile SVR
body: #### What is implemented?

This PR implements quantile regression using support-vector machines.

Mathematically, this applies the "kernel trick" to an L2 regularized linear regression that minimizes the "pinball-loss". For linear kernels and without regularization, this would give the same result as the already existing `QuantileRegressor` (see #9978). The dual problem is derived in Hwang et al. (2005).

Implementation-wise, the algorithm is only a slight modification of epsilon-SVR. In fact, when the quantile is set to 0.5, the regression is exactly equivalent to an epsilon-SVR where epsilon is set to zero! Thanks to this very close similarity, only very few changes are needed w.r.t. epsilon-SVR to make it work. The efficiency is the same as that of epsilon-SVR (for better or worse).

#### Why is this useful?

Scikit-learn already contains a quantile regressor, but it is restricted to solving linear problems. Although these restrictions can partially be alleviated by using transformers into polynomial features or B-Splines, it would be far more useful, especially when dealing with more than one dimension, if one could apply the kernel trick. In addition, L2 regularization is probably more desirable than L1 regularization for most regression problems.

The QuantileSVR regressor can be used to estimate prediction intervals for non-linear functions as shown in the example (see [example code](https://github.com/atrettin/scikit-learn/blob/quantile_svr/examples/svm/plot_svm_quantile.py))
![quantile_svr](https://user-images.githubusercontent.com/11697666/163996807-3b597ff0-e817-443e-9ab9-0229ffaad750.png)

#### References

Hwang, C., Shim, J. (2005). A Simple Quantile Regression via Support Vector Machine. In: Wang, L., Chen, K., Ong, Y.S. (eds) Advances in Natural Computation. ICNC 2005. Lecture Notes in Computer Science, vol 3610. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11539087_66
user: atrettin
issue_23132: state: open
title: Add PAV algorithm for calibration_curve/reliability diagrams
body: ### Describe the workflow you want to enable

```python
import numpy as np
from sklearn.calibration import calibration_curve

y_true = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1])
y_pred = np.array([0.1, 0.2, 0.3, 0.4, 0.65, 0.7, 0.8, 0.9,  1.])
prob_true, prob_pred = calibration_curve(y_true, y_pred, strategy="pav")
```

### Describe your proposed solution

Add the strategy **PAV** as [1] and [2] to `calibration_curve`, there called **CORP**. This is basically applying isotonic regression as binning strategy, which we already have in scikit-learn.

[1] Dimitriadis, T., Gneiting, T., & Jordan, A.I. (2021). Stable reliability diagrams for probabilistic classifiers. Proceedings of the National Academy of Sciences of the United States of America, 118. https://doi.org/10.1073/pnas.2016191118
[2] https://cran.r-project.org/package=reliabilitydiag

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Given the recency of the paper, it has clearly not enough citations (yet). But I have the impression that this is a good strategy for a reliability diagrams with good theoretical and practical properties.

To my knowledge, this strategy is nowhere available in the Python universe as of now.
user: lorentzenchr
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23112: state: open
title: Cache final transformer in pipeline with memory setting
body: ### Describe the bug

When setting the `memory` parameter of a transformer `Pipeline` (i.e., one whose last step is a transformer), the final transformer is not cached.

Discovered at https://stackoverflow.com/q/71812869/10495893.

### Steps/Code to Reproduce

```
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
import time

class Test(BaseEstimator, TransformerMixin):
    def __init__(self, col):
        self.col = col

    def fit(self, X, y=None):
        print(self.col)
        return self

    def transform(self, X, y=None):
        for t in range(5):
            # just to slow it down / check caching.
            print(".")
            time.sleep(1)
        #print(self.col)
        return X

pipline = Pipeline(
    [
        ("test", Test(col="this_column")),
        ("test2", Test(col="that_column"))
    ],
    memory="tmp/cache",
)

pipline.fit(None)
pipline.fit(None)
pipline.fit(None)
```

### Expected Results

```
this_column
.
.
.
.
.
that_column
```

### Actual Results

```
this_column
.
.
.
.
.
that_column
that_column
that_column
```

### Versions

```shell
System:
    python: 3.7.13 (default, Mar 16 2022, 17:37:17)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.4.1
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.2.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

user: bmreiniger
issue_23111: state: open
title: Fix RegressionChain does not accept nans, when base_estimator does #23109
body: Fixes #23109 

user: iuiu34
issue_23107: state: open
title: `SequentialFeatureSelector` is not passing pandas df to estimator/pipeline
body: ### Describe the bug

`SequentialFeatureSelector` cannot be used with a pipeline that expects a pandas dataframe (e.g. a one containing a `ColumnTransformer`) as an input.


At the same time the same pipeline can be used in `cross_val_score`.

### Steps/Code to Reproduce

```
from sklearn.feature_selection import SequentialFeatureSelector
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler
from sklearn.compose import make_column_selector
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

import pandas as pd
import numpy as np

df = pd.DataFrame(data=np.random.rand(10,4), columns=["1","2", "3", "4"])
y = np.random.randint(0,2, size=(10,))

ct = ColumnTransformer(
    transformers=[
        ("numerical", StandardScaler(), make_column_selector(pattern="1")),
    ],
    remainder="passthrough",
)

pipeline = Pipeline([("ct", ct), ("lr", LogisticRegression())])

# this works
print("cross_val_score", cross_val_score(pipeline, df, y, cv=2))

sfs = SequentialFeatureSelector(pipeline, cv=2)
sfs.fit(df, y=y)
```

### Expected Results

No error is thrown and the features are selected.

### Actual Results

```
....sklearn/model_selection/_validation.py:372: FitFailedWarning: 
2 fits failed out of a total of 2.
...
ValueError: make_column_selector can only be applied to pandas dataframes
```

### Versions

```shell
System:
    python: 3.9.5 (default, Nov 23 2021, 15:27:38)  [GCC 9.3.0]
executable: /root/.cache/pypoetry/virtualenvs/science-yd9zGFjU-py3.9/bin/python
   machine: Linux-5.13.0-35-generic-x86_64-with-glibc2.31

Python dependencies:
          pip: 22.0.3
   setuptools: 60.9.3
      sklearn: 1.0.2
        numpy: 1.19.5
        scipy: 1.8.0
       Cython: None
       pandas: 1.4.1
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```
```

user: kretes
issue_23080: state: open
title: Implement Min-Cut clustering
body: ### Describe the workflow you want to enable

Use the exact min-cut culstering method with API similar to sklearn.cluster.SpectralClustering.

### Describe your proposed solution

Implement an algorithm to solve the min-cut problem as a means of dataset clustering. 

### Describe alternatives you've considered, if relevant

In practice SpectralClustering is usally a good alternative. In fact, SpectralClustering can be thought of as an approximate solution to the min-cut problem. 

### Additional context

Solving min-cut exactly is a computationally difficult problem and hence is not practical for large datasets. I think it would be valuable for educational purposes, though (e.g. to compare the performance with SpectralClustering on toy datasets). Furthermore, it may still be useful when working with smaller datasets.   
user: jachymb
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23070: state: open
title: Scikit-learn office hours Monday June 27, 2022 from 4 pm to 5 pm (UTC) on the scikit-learn discord channel
body: Some of us will be online on the scikit-learn discord.

First time and occasional contributors are welcome to join us to
discord using this invitation link:

- https://discord.gg/VVzhr8cHK8

The focus of these office hour sessions is to answer questions about
contributing to scikit-learn. We can also split into break out
audio/text channels and do pair programming or live reviewing of
forgotten pull requests with screen sharing.

We can also try to assist you into crafting minimal reproduction cases
for bug reports to get a higher likelihood of resolution (e.g.
https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

Please note, our Code of Conduct applies:

- https://github.com/scikit-learn/scikit-learn/blob/main/CODE_OF_CONDUCT.md

Here is the link to the calendar for the planned scikit-learn events:

- https://blog.scikit-learn.org/calendar/

See you soon on discord!
user: glemaitre
issue_23065: state: open
title: Propagating missing values for PolynomialFeatures() - Valid Use Case
body: ### Describe the workflow you want to enable

Essentially re-opening #20471 with a valid use-case for sending interaction terms with missing (nan) values to nan:

Consider the case where there's two mutually exclusive binary variables (e.g., count < 10% and count > 10%).  Their interaction term should always be 0, however, if there's missing data and the imputation method results in non-zero values (e.g. mean), the interaction term becomes non-zero in the case where both terms are missing.  Correcting this currently is complicated because we have to propagate this information outside of the pipeline module.  

Essentially, when the imputation method is important for valid feature interpretation, (we work with clinical variable models, so it's very important), sometimes Imputing after makes sense.  It's important, I think, to note that the two operations are not commutative - SimpleImputer(PolynomialFeatures(X)) != PolynomialFeatures(SimpleImputer(X)), and sometimes the latter doesn't result in values that make sense (it's possible to have two different values representing "missing interaction terms").

### Describe your proposed solution

A flag like `propagate_nans` where if true, nan inputs result in nan interaction terms.  Otherwise, the default behavior with an error message.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: aauker
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_23031: state: open
title: ENH Added warning for RidgeCV
body: #### Reference Issues/PRs
Resolves #21303 (stalled)
Resolves #16408 (stalled)
Fixes #16398

#### What does this implement/fix? Explain your changes.
PRs #21303, #16408: Adds warning in `RidgeCV` when the optimal value found for (any, in case `alpha_per_target=True`) `alpha` is at the boundary of the search range.
This PR: Addresses reviewer comments.

#### Any other comments?
user: Micky774
issue_23004: state: open
title: Decouple CountVectorizer => TextTokenizer + ItemCountVectorizer
body: ### Describe the workflow you want to enable

The `CountVectorizer` component has the responsibility of not just vectorizing term frequencies but also internally tokenizing & normalizing the input text into terms. While this functionality is convenient for NLP pipelines, it cannot be leveraged for similar type of non-NLP problems where one needs to perform tf-idf (or similar) on "document" bags of general items other than text.

### Describe your proposed solution

Decouple the `CountVectorizer` into the following 2 specific purpose & reusable components: `TextTokenizer` & `ItemCountVectorizer` such that when both are run sequentially in a pipeline they produce the same behavior.

### Describe alternatives you've considered, if relevant

In some but not all cases one could force this (vectorizing item frequency) behavior by carefully serializing the bag of items into a faked string document & configuring the `CountVectorizer` tokenizer parameters to split on specific delimiters. But this brittle serialization & string de-serialization becomes inefficient & a bit hacky.

### Additional context

_No response_
user: eangius
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_22999: state: open
title: ENH alternative tighter dual gap in elastic net
body: #### Reference Issues/PRs
Fixes #22836.

#### What does this implement/fix? Explain your changes.
This PR adds formulae for the used dual gap.
Additionally, it adds an alternative dual gap to the coordinate descent solver. The alternative gap can potentially be tighter such that the solver can terminate earlier, in particular for higher precision (small `tol`) and late iterations of the coordinate descent solver, see https://github.com/scikit-learn/scikit-learn/issues/22836#issuecomment-1069228270 for some experiments.

#### Any other comments?
Only extends `enet_coordinate_descent`. The other coordinate descent versions are not (yet) modified.
user: lorentzenchr
issue_22966: state: open
title: Out of Fold score calculation for Cross validation
body: ### Describe the workflow you want to enable

What roughly happens in cross-validation as of now: 

```

# k-fold cross validation
scores = list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# evaluate model
	yhat = model.predict(test_X)
	acc = accuracy_score(test_y, yhat)
	# store score

        #------------- NOTE THIS ---------------------
	scores.append(acc)
	print('> ', acc)
# summarize model performance

#------------- AND ALSO NOTE THIS ---------------------
mean_s, std_s = mean(scores), std(scores)
print('Mean: %.3f, Standard Deviation: %.3f' % (mean_s, std_s))
```

Now, this for sure satisfies the purpose. But there should be an option to calculate the OOF score as well. (Described in the solution section) 

### Describe your proposed solution

What OOF would do:

```
data_y, data_yhat = list(), list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# make predictions
	yhat = model.predict(test_X)
	# store
        #------------- NOTE THIS ---------------------
	data_y.extend(test_y)
	data_yhat.extend(yhat)
# evaluate the model

#------------- AND ALSO NOTE THIS ---------------------
acc = accuracy_score(data_y, data_yhat)
print('Accuracy: %.3f' % (acc))
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

An article on OOF predictions and score - https://machinelearningmastery.com/out-of-fold-predictions-in-machine-learning/
user: aadarshsingh191198
issue_22947: state: open
title: BUG unpenalized Ridge does not give minimum norm solution
body: #### Describe the bug
As noted in #22910, `Ridge(alpha=0, fit_intercept=True)` does not give the minimal norm solution for wide data, i.e. `n_features > n_samples`.

Note that we nowhere guarantee that we provide the **minimum norm solution**.

#### Probable Cause
For wide X, the least squares problem reads a bit different: $\mathrm{min} ||w||_2$ subject to $Xw = y$ with solution $w = X'(XX')^{-1} y$, see e.g. http://ee263.stanford.edu/lectures/min-norm.pdf.
With explicit intercept $w_0$, this reads $w = X'(XX' + 1 1')^{-1} y$, where 1 is a column vector of ones. $w_0 = 1'(XX' + 1 1')^{-1} y$.
**This is incompatible with our mean centering approach.**

#### Example

<details>

```python
import numpy as np
from numpy.testing import assert_allclose
from scipy import linalg
from sklearn.datasets import make_low_rank_matrix
from sklearn.linear_model import Ridge

n_samples, n_features = 4, 12  # wide data
k = min(n_samples, n_features)
rng = np.random.RandomState(42)
X = make_low_rank_matrix(
    n_samples=n_samples, n_features=n_features, effective_rank=k
)
X[:, -1] = 1  # last columns acts as intercept
U, s, Vt = linalg.svd(X)
assert np.all(s) > 1e-3  # to be sure X is not singular
U1, U2 = U[:, :k], U[:, k:]
Vt1, _ = Vt[:k, :], Vt[k:, :]
y = rng.uniform(low=-10, high=10, size=n_samples)
# w = X'(XX')^-1 y = V s^-1 U' y
coef_ols = Vt1.T @ np.diag(1 / s) @ U1.T @ y

model = Ridge(alpha=0, fit_intercept=True)
X = X[:, :-1]  # remove intercept
intercept = coef_ols[-1]
coef = coef_ols[:-1]
model.fit(X, y)

# Check that we have found a solution => residuals = 0
assert_allclose(model.predict(X), y)
# Check that `coef`, `intercept` also provide a valid solution
assert_allclose(X @ coef + intercept, y)

# Ridge does not give the minimum norm solution. (This should be equal.)
np.linalg.norm(np.r_[model.intercept_, model.coef_]) > np.linalg.norm(
    np.r_[intercept, coef]
)
```
This last statement should be be `False`. It proves that Ridge does not give the miminum norm solution. 

</details>
user: lorentzenchr
issue_22945: state: open
title: GaussianProcessRegressor (predict)
body: ### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/22925

<div type='discussions-op-text'>

<sup>Originally posted by **jecampagne** March 23, 2022</sup>
Hello,
I am questioning the code of `predict` of the  `GaussianProcessRegressor`. The code is based on Alg. 2.1 of C. E. Rasmussen & C. K. I. Williams (2003). I have the 2006 version and I do not know if there have been a modification between the two versions. Well, the algorithm is based on the following formula:

![image](https://user-images.githubusercontent.com/20539759/159709115-293c9cc3-aaf9-497f-9c0a-a7c9c10da73f.png)

![image](https://user-images.githubusercontent.com/20539759/159711732-5c3b305a-ecb0-4874-b79b-11680eb670a5.png)


Notice that the `K(X,X)` (ie, X=X_train) is the only one that contains the "noise" parameter while `K(X*, X)` and `K(X*,X*)` (ie, X*=X_test) do not get this additional part to the diagonal. And this is ok.

Looking now at the code it is ok for the default kernel (ie. RBF with fixed scale and length):
```python
self.kernel_ = C(1.0, constant_value_bounds="fixed") * RBF(
                1.0, length_scale_bounds="fixed"
            )
```
But, if the user pass a kernel composed with the `WhiteKernel`as
```python
kernel = C(1.0) * RBF() + WhiteKernel(0.5)
```
then it seems that `K(X*, X)` and `K(X*,X*)` will use the `WhiteKernel` part which is not what the Alg. 2.1 of Rasmussen & Williams does.




</div>
user: jecampagne
issue_22923: state: open
title: ENH Support cardinality filtering of columns in `make_column_selector`
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Resolves #15873.

#### What does this implement/fix? Explain your changes.
Implemented the functionality in the PR (selecting columns with cardinality > or <= some threshold), and a test in the test suite to check it works as expected.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: richardt94
issue_22914: state: open
title: Calculation of alphas in ElasticNetCV doesn't use sample_weight
body: ### Describe the bug

In ElasticNetCV, the first and largest value of `alpha`, call it `alpha_max`, should be just large enough to force all of the coefficients to become zero. The existing code works correctly when `sample_weight` is not specified. However, the computation of `alpha_max` does not take into account `sample_weight`.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.linear_model import ElasticNet, ElasticNetCV

X = np.array([[3, 1], [2, 5], [5, 3], [1, 4]])
beta = np.array([1, 1])
y = X @ beta
w = np.array([10, 1, 10, 1])

# Fit ElasticNetCV just to get the .alphas_ attribute
enetCV = ElasticNetCV(cv=2)
enetCV.fit(X, y, sample_weight=w)

# The coefficient of ElasticNet fitted at alpha_max should be [0.  0.].
alpha_max = enetCV.alphas_[0]
enet = ElasticNet(alpha=alpha_max)
enet.fit(X, y, sample_weight=w)
print(enet.coef_)  # [0.1970807  0.19708023]
```

### Expected Results

If the correct value of `alpha_max` is computed, then `enet.coef_` should be right at the cusp of zero, such that any smaller value of `alpha` makes it nonzero:
```python
def get_alpha_max(X, y, w, l1_ratio=0.5):
    wn = w / w.sum()
    Xn = X - np.dot(wn, X)
    yn = (y - np.dot(wn, y)) * wn
    return np.max(np.abs(yn @ Xn)) / l1_ratio


enet = ElasticNet(alpha=get_alpha_max(X, y, w))
enet.fit(X, y, sample_weight=w)
print(enet.coef_)  # [6.70427878e-17 6.70427878e-17]
```

### Actual Results

`enet.coef_` is `[0.1970807  0.19708023]`.

### Versions

```shell
System:
    python: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0]
executable: /home/jhopfens/.conda/envs/jhop39/bin/python
   machine: Linux-3.10.0-1160.53.1.el7.x86_64-x86_64-with-glibc2.17

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.7.2
       Cython: 0.29.24
       pandas: 1.3.5
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.0.0

Built with OpenMP: True
```

user: s-banach
issue_22893: state: open
title: SLEP006 - Metadata Routing task list
body: This issue is to track the work we need to do before we can merge `sample-props` branch into `main`:

- [x] Based on this prototype: https://github.com/scikit-learn/scikit-learn/pull/16079
- [x] Merge https://github.com/scikit-learn/scikit-learn/pull/22083 into `sample-props`. This PR only touches `BaseEstimator` and hence _consumers_. It does NOT touch meta-estimators, scorers or cv splitters.
- [x] Work on splitters and merge that into `sample-props`: https://github.com/scikit-learn/scikit-learn/pull/22765
- [x] feature flag: https://github.com/scikit-learn/scikit-learn/issues/26045: https://github.com/scikit-learn/scikit-learn/pull/26103
- [x] Add new terms to the glossary: router, consumer, metadata: https://github.com/scikit-learn/scikit-learn/pull/26685
- [x] Work on scorers and merge that into `sample-props` ~(note that this involves an ongoing discussion on whether we'd like to mutate a scorer or not)~: https://github.com/scikit-learn/scikit-learn/pull/22757
- [x] Get backward compatibility mechanisms in place for meta-estimators which already route given metadata: https://github.com/scikit-learn/scikit-learn/pull/22986
- [ ] Work on meta-estimators: this involves writing `get_metadata_routing` in easy cases, and a whole lot more in cases where we'd like to keep backward compatibility in parsing input args suchs as `estimator__param` in `Pipeline`
- [ ] Once all the above is merged into `sample-props`, do a few tests with third party estimators to see if they'd work out of the box. Note that _consumer_ estimators should work out of the box as long as they inherit from `BaseEstimator` and their `fit` accepts metadata as explicit arguments rather than `**kwargs`.
- [ ] Check whether a library such as cuML could vendor `_metadata_requests.py` and work with scikit-learn meta-estimators w/o depending on the library.
- [x] Refactor tests: https://github.com/scikit-learn/scikit-learn/issues/23918
- [x] Merge `sample-props` into `main`: https://github.com/scikit-learn/scikit-learn/pull/24027

## Enhancements:
- [x] https://github.com/scikit-learn/scikit-learn/pull/24023
- [ ] https://github.com/scikit-learn/scikit-learn/issues/18936

## Cleanup
- [ ] https://github.com/scikit-learn/scikit-learn/issues/26505
	- [x] https://github.com/scikit-learn/scikit-learn/pull/26503

## Open issues:
- [x] https://github.com/scikit-learn/scikit-learn/issues/22987
	- [x] https://github.com/scikit-learn/scikit-learn/pull/26506
- [x] https://github.com/scikit-learn/scikit-learn/issues/22988: https://github.com/scikit-learn/scikit-learn/pull/23342
- [ ] https://github.com/scikit-learn/scikit-learn/issues/23920
- [ ] https://github.com/scikit-learn/scikit-learn/issues/23928
	- [ ] global flags: https://github.com/scikit-learn/scikit-learn/issues/26050
	- [ ] auto-routing: https://github.com/scikit-learn/scikit-learn/issues/26179
- [x] https://github.com/scikit-learn/scikit-learn/issues/23933
	- [x] https://github.com/scikit-learn/scikit-learn/pull/26911
- [x] https://github.com/scikit-learn/scikit-learn/issues/25776: https://github.com/scikit-learn/scikit-learn/pull/25852 - resolution is to use the feature flag instead of breaking change.
- [ ] https://github.com/scikit-learn/scikit-learn/issues/27977

Our plan is to hopefully have this feature in 1.1, which we should be releasing in late April/early May.

Here's a list of meta-estimators which need to be updated:
- [ ] AdaBoostClassifier: https://github.com/scikit-learn/scikit-learn/pull/24026
- [ ] AdaBoostRegressor: https://github.com/scikit-learn/scikit-learn/pull/24026
- [ ] BaggingClassifier: https://github.com/scikit-learn/scikit-learn/pull/24250, https://github.com/scikit-learn/scikit-learn/pull/28432
- [ ] BaggingRegressor: https://github.com/scikit-learn/scikit-learn/pull/24250, https://github.com/scikit-learn/scikit-learn/pull/28432
- [x] CalibratedClassifierCV: https://github.com/scikit-learn/scikit-learn/pull/24126
- [x] ClassifierChain: #24443
- [x] ColumnTransformer: https://github.com/scikit-learn/scikit-learn/pull/27005
- [x] ElasticNetCV: https://github.com/scikit-learn/scikit-learn/pull/27478 
- [ ] FeatureUnion
- [ ] GraphicalLassoCV: https://github.com/scikit-learn/scikit-learn/pull/27566
- [x] GridSearchCV, HalvingGridSearchCV, HalvingRandomSearchCV, RandomizedSearchCV: #27058
- [ ] IterativeImputer: https://github.com/scikit-learn/scikit-learn/pull/28187
- [x] LarsCV: https://github.com/scikit-learn/scikit-learn/pull/27538
- [x] LassoCV: https://github.com/scikit-learn/scikit-learn/pull/27478
- [x] LassoLarsCV: https://github.com/scikit-learn/scikit-learn/pull/27538
- [x] LogisticRegressionCV https://github.com/scikit-learn/scikit-learn/pull/26525
- [x] MultiOutputClassifier https://github.com/scikit-learn/scikit-learn/pull/22986
- [x] MultiOutputRegressor https://github.com/scikit-learn/scikit-learn/pull/22986
- [x] MultiTaskElasticNetCV: https://github.com/scikit-learn/scikit-learn/pull/27478
- [x] MultiTaskLassoCV: https://github.com/scikit-learn/scikit-learn/pull/27478
- [x] OneVsOneClassifier, OneVsRestClassifier, OutputCodeClassifier: https://github.com/scikit-learn/scikit-learn/pull/27308
- [x] OrthogonalMatchingPursuitCV: https://github.com/scikit-learn/scikit-learn/pull/27500
- [x] Pipeline: https://github.com/scikit-learn/scikit-learn/pull/26789
- [ ] RANSACRegressor
- [ ] RFE
- [ ] RFECV
- [x] RegressorChain: #24443
- [ ] RidgeCV: https://github.com/scikit-learn/scikit-learn/pull/27560
- [ ] RidgeClassifierCV: https://github.com/scikit-learn/scikit-learn/pull/27560
- [ ] SelectFromModel
    - [ ] https://github.com/scikit-learn/scikit-learn/pull/27490
- [ ] SelfTrainingClassifier
- [ ] SequentialFeatureSelector
- [ ] StackingClassifier
- [ ] StackingRegressor
- [ ] TransformedTargetRegressor
- [ ] VotingClassifier
- [ ] VotingRegressor
- [x] cross_validate, cross_val_score, cross_val_predict
	- [x] https://github.com/scikit-learn/scikit-learn/pull/26896
- [ ] permutation_test_score
- [ ] learning_curve
- [ ] validation_curve

cc @jnothman @thomasjpfan @lorentzenchr 

## Misc:

- https://github.com/scikit-learn/scikit-learn/pull/25851
- https://github.com/scikit-learn/scikit-learn/pull/27389

-------------------------------------------------------------
From #24027 

Once we're finished with merging PRs to `sample-props` branch, we should merge this.

I've opened this for us to have an idea of what the whole project is touching.

We probably should rebase and merge this instead of squash and merge.

This closes and fixes the following issues and PRs:

- Routing issues:
Enables #6322, 

- AdaBoost:
Fixes #2630, 
Fixes #21706

- ClassifierChain:
Fixes  #11429, 

- FeatureUnion:
Fixes #7136

- LogisticRegressionCV:
Fixes  #8950,

- Multioutput*
Fixes #15953

- OneVsRestClassifier
Fixes #10882

- RFE/RFECV:
Fixes #7308

- *SearchCV:
Fixes #8127, 
Fixes #8158,

- cross_val*:
Fixes #13432, 
Fixes #4632, 
Fixes #7646, 
Fixes #20349

- Stacking*
Fixes #18028,

- Voting*
Fixes #20167

user: adrinjalali
labels: Hard: Hard level of difficulty
Meta-issue: General issue associated to an identified list of tasks
issue_22881: state: open
title: Improve tests to make them run on variously typed data using the `global_dtype` fixture
body: ## Context: the new `global_dtype` fixture and `SKLEARN_RUN_FLOAT32_TESTS` environment variable

Introduction of low-level computational routines for 32bit motivated an extension of tests to run them on 32bit.

In this regards, https://github.com/scikit-learn/scikit-learn/pull/22690 introduced a new `global_dtype` fixture as well has the `SKLEARN_RUN_FLOAT32_TESTS` env. variable to make it possible to run the test on 32bit data.

Running test on 32bit can be done using `SKLEARN_RUN_FLOAT32_TESTS=1`.

For instance, this run the first `global_dtype`-parametrised test:

```
SKLEARN_RUN_FLOAT32_TESTS=1 pytest sklearn/feature_selection/tests/test_mutual_info.py -k test_compute_mi_cc
```

This allows running tests on 32bit dataset on some CI job, and currently a single CI job is used to run tests on 32bit.

More details about the fixture in the online dev doc for the `SKLEARN_TESTS_GLOBAL_RANDOM_SEED` env variable:

https://scikit-learn.org/dev/computing/parallelism.html#environment-variables

## Guidelines to convert existing tests

- Not all scikit-learn tests must use this fixture. We must parametrise tests that actually assert closeness of results using `assert_allclose`. For instance, tests that check for the exception messages raised when passing invalid inputs must not be converted.

- Tests using `np.testing.assert_allclose` must now use [`sklearn.utils._testing.assert_allclose`](https://github.com/scikit-learn/scikit-learn/blob/613773db11a32cf3acc87f3d33a8139d03707baf/sklearn/utils/_testing.py#L390) as a drop-in replacement.

- Check that the dtype of fitted attributes or return values that depend on the dtype of the input datastructure actually have the expected dtype: typically when all inputs are continuous values in float32, it is often (but not always) the case that scikit-learn should carry all arithmetic operations at that precision level and return output arrays with the same precision level. There can be exceptions, in which case they could be made explicit with an inline comment in the test, possibly with a `TODO` marker when one thing that the current behavior should change (see the related: #11000 and #22682 for instance).

- To avoid having to review huge PRs that impact many files at once and can lead to conflicts, let's open PRs that edit at most one test file at a time. For instance use a title such as:

> TST use global_dtype in sklearn/_loss/tests/test_glm_distribution.py

- Please reference #22881 (i.e. this issue) in the description of the PR and put the full filename of the test file you edit in the title of the PR.

- To convert an existing test with a fixed dtype, the general pattern is to rewrite a function such as:

```python
from numpy.testing import assert_allclose

def test_some_function():
    # ...
    rng = np.random.RandomState(0)
    X = rng.randon.rand(n_samples, n_features)
    y = rng.randon.rand(n_samples).astype(global_dtype)
    model.fit(X, y)
    # ...
    y_pred = model.predict(X)
    assert_allclose(y_pred, y_true)
```

to:

```python
from sklearn.utils._testing import assert_allclose

def test_some_function(global_dtype):
    # ...
    rng = np.random.RandomState(0)
    X = rng.randon.rand(n_samples, n_features).astype(global_dtype, copy=False)
    y = rng.randon.rand(n_samples).astype(global_dtype, copy=False)
    model.fit(X, y)
    # ...
    assert model.fitted_param_.dtype == global_dtype
    y_pred = model.predict(X)
    assert y_pred.dtype == global_dtype
    assert_allclose(y_pred, y_true)
```

and then check that the test is passing on 32bit datasets
```
SKLEARN_RUN_FLOAT32_TESTS=1 pytest sklearn/some_module/test/test_some_module.py -k test_some_function
```

Failures are to be handle on a case-by-case basis.

## List of test modules to upgrade

```
find sklearn -name "test_*.py"
```

- [ ] sklearn/_loss/tests/test_glm_distribution.py
- [ ] sklearn/_loss/tests/test_link.py
- [ ] sklearn/_loss/tests/test_loss.py
- [ ] sklearn/cluster/tests/test_affinity_propagation.py #22667
- [ ] sklearn/cluster/tests/test_bicluster.py
- [ ] sklearn/cluster/tests/test_birch.py #22671
- [ ] sklearn/cluster/tests/test_dbscan.py
- [ ] sklearn/cluster/tests/test_feature_agglomeration.py
- [ ] sklearn/cluster/tests/test_hierarchical.py
- [ ] sklearn/cluster/tests/test_k_means.py
- [x] sklearn/cluster/tests/test_mean_shift.py #22672
- [x] sklearn/cluster/tests/test_optics.py #22665
- [ ] sklearn/cluster/tests/test_spectral.py #22669
- [ ] sklearn/compose/tests/test_column_transformer.py
- [ ] sklearn/compose/tests/test_target.py
- [ ] sklearn/covariance/tests/test_covariance.py
- [ ] sklearn/covariance/tests/test_elliptic_envelope.py
- [ ] sklearn/covariance/tests/test_graphical_lasso.py
- [ ] sklearn/covariance/tests/test_robust_covariance.py
- [ ] sklearn/cross_decomposition/tests/test_pls.py
- [ ] sklearn/datasets/tests/test_20news.py
- [ ] sklearn/datasets/tests/test_base.py
- [ ] sklearn/datasets/tests/test_california_housing.py
- [ ] sklearn/datasets/tests/test_common.py
- [ ] sklearn/datasets/tests/test_covtype.py
- [ ] sklearn/datasets/tests/test_kddcup99.py
- [ ] sklearn/datasets/tests/test_lfw.py
- [ ] sklearn/datasets/tests/test_olivetti_faces.py
- [ ] sklearn/datasets/tests/test_openml.py
- [ ] sklearn/datasets/tests/test_rcv1.py
- [ ] sklearn/datasets/tests/test_samples_generator.py
- [ ] sklearn/datasets/tests/test_svmlight_format.py
- [ ] sklearn/decomposition/tests/test_dict_learning.py
- [ ] sklearn/decomposition/tests/test_factor_analysis.py
- [ ] sklearn/decomposition/tests/test_fastica.py
- [ ] sklearn/decomposition/tests/test_incremental_pca.py
- [ ] sklearn/decomposition/tests/test_kernel_pca.py
- [ ] sklearn/decomposition/tests/test_nmf.py
- [ ] sklearn/decomposition/tests/test_online_lda.py
- [ ] sklearn/decomposition/tests/test_pca.py
- [ ] sklearn/decomposition/tests/test_sparse_pca.py
- [ ] sklearn/decomposition/tests/test_truncated_svd.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_binning.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_bitset.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_compare_lightgbm.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_grower.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_histogram.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_monotonic_contraints.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_predictor.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_splitting.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_warm_start.py
- [ ] sklearn/ensemble/tests/test_bagging.py
- [ ] sklearn/ensemble/tests/test_base.py
- [ ] sklearn/ensemble/tests/test_common.py
- [ ] sklearn/ensemble/tests/test_forest.py
- [ ] sklearn/ensemble/tests/test_gradient_boosting_loss_functions.py
- [ ] sklearn/ensemble/tests/test_gradient_boosting.py
- [ ] sklearn/ensemble/tests/test_iforest.py
- [ ] sklearn/ensemble/tests/test_stacking.py
- [ ] sklearn/ensemble/tests/test_voting.py
- [ ] sklearn/ensemble/tests/test_weight_boosting.py
- [ ] sklearn/experimental/tests/test_enable_hist_gradient_boosting.py
- [ ] sklearn/experimental/tests/test_enable_iterative_imputer.py
- [ ] sklearn/experimental/tests/test_enable_successive_halving.py
- [ ] sklearn/feature_extraction/tests/test_dict_vectorizer.py
- [ ] sklearn/feature_extraction/tests/test_feature_hasher.py
- [ ] sklearn/feature_extraction/tests/test_image.py
- [ ] sklearn/feature_extraction/tests/test_text.py
- [ ] sklearn/feature_selection/tests/test_base.py
- [ ] sklearn/feature_selection/tests/test_chi2.py
- [ ] sklearn/feature_selection/tests/test_feature_select.py
- [ ] sklearn/feature_selection/tests/test_from_model.py
- [x] sklearn/feature_selection/tests/test_mutual_info.py #22677
- [ ] sklearn/feature_selection/tests/test_rfe.py
- [ ] sklearn/feature_selection/tests/test_sequential.py
- [ ] sklearn/feature_selection/tests/test_variance_threshold.py
- [ ] sklearn/gaussian_process/tests/test_gpc.py
- [ ] sklearn/gaussian_process/tests/test_gpr.py
- [ ] sklearn/gaussian_process/tests/test_kernels.py
- [ ] sklearn/impute/tests/test_base.py
- [ ] sklearn/impute/tests/test_common.py
- [ ] sklearn/impute/tests/test_impute.py
- [ ] sklearn/impute/tests/test_knn.py
- [ ] sklearn/inspection/_plot/tests/test_plot_partial_dependence.py
- [ ] sklearn/inspection/tests/test_partial_dependence.py
- [ ] sklearn/inspection/tests/test_permutation_importance.py
- [ ] sklearn/linear_model/_glm/tests/test_glm.py
- [ ] sklearn/linear_model/_glm/tests/test_link.py
- [ ] sklearn/linear_model/tests/test_base.py
- [ ] sklearn/linear_model/tests/test_bayes.py
- [ ] sklearn/linear_model/tests/test_common.py
- [ ] sklearn/linear_model/tests/test_coordinate_descent.py
- [ ] sklearn/linear_model/tests/test_huber.py
- [ ] sklearn/linear_model/tests/test_least_angle.py
- [ ] sklearn/linear_model/tests/test_linear_loss.py
- [ ] sklearn/linear_model/tests/test_logistic.py
- [ ] sklearn/linear_model/tests/test_omp.py
- [ ] sklearn/linear_model/tests/test_passive_aggressive.py
- [ ] sklearn/linear_model/tests/test_perceptron.py
- [ ] sklearn/linear_model/tests/test_quantile.py
- [ ] sklearn/linear_model/tests/test_ransac.py
- [ ] sklearn/linear_model/tests/test_ridge.py
- [ ] sklearn/linear_model/tests/test_sag.py
- [ ] sklearn/linear_model/tests/test_sgd.py
- [ ] sklearn/linear_model/tests/test_sparse_coordinate_descent.py
- [ ] sklearn/linear_model/tests/test_theil_sen.py
- [x] sklearn/manifold/tests/test_isomap.py #22673
- [x] sklearn/manifold/tests/test_locally_linear.py #22676
- [ ] sklearn/manifold/tests/test_mds.py
- [ ] sklearn/manifold/tests/test_spectral_embedding.py
- [ ] sklearn/manifold/tests/test_t_sne.py #22675
- [ ] sklearn/metrics/_plot/tests/test_base.py
- [ ] sklearn/metrics/_plot/tests/test_common_curve_display.py
- [ ] sklearn/metrics/_plot/tests/test_confusion_matrix_display.py
- [ ] sklearn/metrics/_plot/tests/test_det_curve_display.py
- [ ] sklearn/metrics/_plot/tests/test_plot_confusion_matrix.py
- [ ] sklearn/metrics/_plot/tests/test_plot_curve_common.py
- [ ] sklearn/metrics/_plot/tests/test_plot_det_curve.py
- [ ] sklearn/metrics/_plot/tests/test_plot_precision_recall.py
- [ ] sklearn/metrics/_plot/tests/test_plot_roc_curve.py
- [ ] sklearn/metrics/_plot/tests/test_precision_recall_display.py
- [ ] sklearn/metrics/_plot/tests/test_roc_curve_display.py
- [ ] sklearn/metrics/cluster/tests/test_bicluster.py
- [ ] sklearn/metrics/cluster/tests/test_common.py
- [ ] sklearn/metrics/cluster/tests/test_supervised.py
- [ ] sklearn/metrics/cluster/tests/test_unsupervised.py
- [ ] sklearn/metrics/tests/test_classification.py
- [ ] sklearn/metrics/tests/test_common.py
- [ ] sklearn/metrics/tests/test_dist_metrics.py
- [ ] sklearn/metrics/tests/test_pairwise_distances_reduction.py
- [x] sklearn/metrics/tests/test_pairwise.py #22666
- [ ] sklearn/metrics/tests/test_ranking.py
- [ ] sklearn/metrics/tests/test_regression.py
- [ ] sklearn/metrics/tests/test_score_objects.py
- [ ] sklearn/mixture/tests/test_bayesian_mixture.py
- [ ] sklearn/mixture/tests/test_gaussian_mixture.py
- [ ] sklearn/mixture/tests/test_mixture.py
- [ ] sklearn/model_selection/tests/test_search.py
- [ ] sklearn/model_selection/tests/test_split.py
- [ ] sklearn/model_selection/tests/test_successive_halving.py
- [ ] sklearn/model_selection/tests/test_validation.py
- [ ] sklearn/neighbors/tests/test_ball_tree.py
- [ ] sklearn/neighbors/tests/test_graph.py
- [ ] sklearn/neighbors/tests/test_kd_tree.py
- [ ] sklearn/neighbors/tests/test_kde.py
- [ ] sklearn/neighbors/tests/test_lof.py #22665
- [ ] sklearn/neighbors/tests/test_nca.py
- [ ] sklearn/neighbors/tests/test_nearest_centroid.py
- [ ] sklearn/neighbors/tests/test_neighbors_pipeline.py
- [ ] sklearn/neighbors/tests/test_neighbors_tree.py
- [x] sklearn/neighbors/tests/test_neighbors.py #22663
- [ ] sklearn/neighbors/tests/test_quad_tree.py
- [ ] sklearn/neural_network/tests/test_base.py
- [ ] sklearn/neural_network/tests/test_mlp.py
- [ ] sklearn/neural_network/tests/test_rbm.py
- [ ] sklearn/neural_network/tests/test_stochastic_optimizers.py
- [ ] sklearn/preprocessing/tests/test_common.py
- [ ] sklearn/preprocessing/tests/test_data.py
- [ ] sklearn/preprocessing/tests/test_discretization.py
- [ ] sklearn/preprocessing/tests/test_encoders.py
- [ ] sklearn/preprocessing/tests/test_function_transformer.py
- [ ] sklearn/preprocessing/tests/test_label.py
- [ ] sklearn/preprocessing/tests/test_polynomial.py
- [ ] sklearn/semi_supervised/tests/test_label_propagation.py
- [ ] sklearn/semi_supervised/tests/test_self_training.py
- [ ] sklearn/svm/tests/test_bounds.py
- [ ] sklearn/svm/tests/test_sparse.py
- [ ] sklearn/svm/tests/test_svm.py
- [ ] sklearn/tests/test_base.py
- [ ] sklearn/tests/test_build.py
- [ ] sklearn/tests/test_calibration.py
- [ ] sklearn/tests/test_check_build.py
- [ ] sklearn/tests/test_common.py
- [ ] sklearn/tests/test_config.py
- [ ] sklearn/tests/test_discriminant_analysis.py
- [ ] sklearn/tests/test_docstring_parameters.py
- [ ] sklearn/tests/test_docstrings.py
- [ ] sklearn/tests/test_dummy.py
- [ ] sklearn/tests/test_init.py
- [ ] sklearn/tests/test_isotonic.py
- [ ] sklearn/tests/test_kernel_approximation.py
- [ ] sklearn/tests/test_kernel_ridge.py
- [ ] sklearn/tests/test_metaestimators.py
- [ ] sklearn/tests/test_min_dependencies_readme.py
- [ ] sklearn/tests/test_multiclass.py
- [ ] sklearn/tests/test_multioutput.py
- [ ] sklearn/tests/test_naive_bayes.py
- [ ] sklearn/tests/test_pipeline.py
- [ ] sklearn/tests/test_random_projection.py
- [ ] sklearn/tree/tests/test_export.py
- [ ] sklearn/tree/tests/test_reingold_tilford.py
- [ ] sklearn/tree/tests/test_tree.py
- [ ] sklearn/utils/tests/test_arpack.py
- [ ] sklearn/utils/tests/test_arrayfuncs.py
- [ ] sklearn/utils/tests/test_class_weight.py
- [ ] sklearn/utils/tests/test_cython_blas.py
- [ ] sklearn/utils/tests/test_cython_templating.py
- [ ] sklearn/utils/tests/test_deprecation.py
- [ ] sklearn/utils/tests/test_encode.py
- [ ] sklearn/utils/tests/test_estimator_checks.py
- [ ] sklearn/utils/tests/test_estimator_html_repr.py
- [ ] sklearn/utils/tests/test_extmath.py
- [ ] sklearn/utils/tests/test_fast_dict.py
- [ ] sklearn/utils/tests/test_fixes.py
- [ ] sklearn/utils/tests/test_graph.py
- [ ] sklearn/utils/tests/test_metaestimators.py
- [ ] sklearn/utils/tests/test_mocking.py
- [ ] sklearn/utils/tests/test_multiclass.py
- [ ] sklearn/utils/tests/test_murmurhash.py
- [ ] sklearn/utils/tests/test_optimize.py
- [ ] sklearn/utils/tests/test_parallel.py
- [ ] sklearn/utils/tests/test_pprint.py
- [ ] sklearn/utils/tests/test_random.py
- [ ] sklearn/utils/tests/test_readonly_wrapper.py
- [ ] sklearn/utils/tests/test_seq_dataset.py
- [ ] sklearn/utils/tests/test_shortest_path.py
- [ ] sklearn/utils/tests/test_show_versions.py
- [ ] sklearn/utils/tests/test_sparsefuncs.py
- [ ] sklearn/utils/tests/test_stats.py
- [ ] sklearn/utils/tests/test_tags.py
- [ ] sklearn/utils/tests/test_testing.py
- [ ] sklearn/utils/tests/test_utils.py
- [ ] sklearn/utils/tests/test_validation.py
- [ ] sklearn/utils/tests/test_weight_vector.py

Note that some of those files might not have any test to update.

user: jjerphan
labels: Hard: Hard level of difficulty
module:test-suite: everything related to our tests
Meta-issue: General issue associated to an identified list of tasks
float32: Issues related to support for 32bit data
issue_22855: state: open
title: The fit performance of LinearRegression is sub-optimal
body: It seems that the performance of Linear Regression is sub-optimal when the number of samples is very large. 

[sklearn_benchmarks](https://github.com/mbatoul/sklearn_benchmarks) measures a [speedup of 48](https://mbatoul.github.io/sklearn_benchmarks/results/github_ci/master/20220315T132336/scikit_learn_intelex_vs_scikit_learn.html#linearregression) compared to an optimized implementation from scikit-learn-intelex on a `1000000x100` dataset. For a given set of parameters and a given dataset, we compute the speed-up `time scikit-learn` / `time sklearnex`. A speed-up of 48 means that sklearnex is 48 times faster than scikit-learn on the given dataset.

[Profiling](https://mbatoul.github.io/sklearn_benchmarks/results/github_ci/improve_cli/20220314T105014/profiling/sklearn_fit_0c037d7431f8fb8e89db7b3f3f29b083_ddd328af83cd4ef15164ddaa0fa01ba4.json.gz) allows a more detailed analysis of the execution of the algorithm. We observe that most of the execution time is spent in the [`lstsq` solver of scipy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.lstsq.html).

<img width="1538" alt="image" src="https://user-images.githubusercontent.com/36322423/158272733-92f107f4-92cf-4e26-887d-2a0818113d2d.png">

The profiling reports of sklearn_benchmarks can be viewed with [Perfetto UI](https://ui.perfetto.dev/).

<details>
  <summary>See benchmark environment information</summary>
  
<img width="946" alt="image" src="https://user-images.githubusercontent.com/36322423/158276716-2ab08b7d-079d-4cb8-82fc-054e6b9e8f40.png">
</details>

It seems that the solver could be better chosen when the number of samples is very large. Perhaps Ridge's solver with a zero penalty could be chosen in this case. On the same dimensions, [it shows better performance](https://mbatoul.github.io/sklearn_benchmarks/results/github_ci/improve_cli/20220314T105014/scikit_learn_intelex_vs_scikit_learn.html#Ridge).

Speedups can be reproduced with the following code:

```bash
conda create -n lr_perf -c conda-forge scikit-learn scikit-learn-intelex numpy jupyter
conda activate lr_perf
```

```python
from sklearn.linear_model import LinearRegression as LinearRegressionSklearn
from sklearnex.linear_model import LinearRegression as LinearRegressionSklearnex
from sklearn.datasets import make_regression
import time
import numpy as np

X, y = make_regression(n_samples=1_000_000, n_features=100, n_informative=10)

def measure(estimator, X, y, n_executions=10):
    times = []
    while len(times) < n_executions:
        t0 = time.perf_counter()
        estimator.fit(X, y)
        t1 = time.perf_counter()
        times.append(t1 - t0)

    return np.mean(times)

mean_time_sklearn = measure(
    estimator=LinearRegressionSklearn(),
    X=X,
    y=y
)

mean_time_sklearnex = measure(
    estimator=LinearRegressionSklearnex(),
    X=X,
    y=y
)

speedup = mean_time_sklearn / mean_time_sklearnex
speedup
```

user: mbatoul
issue_22854: state: open
title: Birch is not finding clusters correctly
body: ### Describe the bug

Birch do not correctly find clusters if the number of samples of one cluster is much larger than another cluster even if the distance between clusters is much larger than threshold.

### Steps/Code to Reproduce

```python
from sklearn.datasets import make_blobs
from sklearn.cluster import Birch
import numpy as np

X1, y1 = make_blobs(n_samples=10000, centers=[[0.6, 0.6]], cluster_std=0.01, random_state=0)
X2, y2 = make_blobs(n_samples=100, centers=[[0.1, 0.1]], cluster_std=0.01, random_state=0)

X = np.append(X1, X2, axis = 0)

brc = Birch(n_clusters=None, threshold = 0.1)
brc.fit(X)

brc.subcluster_centers_
```

### Expected Results

Expected two subclusters  since the distance between clusters is much larger than used threshold.

### Actual Results

Only one subcluster is detected

`brc.subcluster_centers_`:

```
array([[0.59494558, 0.59509375]])
```


### Versions

```shell
System:
    python: 3.8.6 | packaged by conda-forge | (default, Jan 25 2021, 23:21:18)  [GCC 9.3.0]
executable: /home/envs/env1_custom/bin/python
   machine: Linux-4.14.252

Python dependencies:
          pip: 22.0.3
   setuptools: 60.9.3
      sklearn: 1.0.2
        numpy: 1.20.2
        scipy: 1.7.0
       Cython: None
       pandas: 1.2.4
   matplotlib: 3.5.1
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```

user: alhaal
issue_22849: state: open
title: Verbosity option is not working in GridSearchCV (Jupyter notebook)
body: ### Describe the bug

So this issue has been addressed before [here ](https://github.com/scikit-learn/scikit-learn/issues/22291) by [darrencl](https://github.com/darrencl), but the user didn't follow up with [lesteve](https://github.com/lesteve) response.

The problem is that GridSearchCV doesn't show the elapsed time periodically, or any log, I am set`n_jobs = -1`, and `verbose = 1`. I tried setting `n_jobs` to other values, the same with `verbose`, but nothing happened.
Note that this didn't happen until I updated scikit-learn from version 0.22.1 to 1.0.2.

[lesteve](https://github.com/lesteve) in his response assumed that this problem is due to `ipykernel <6`, which is not the case with me.

### Steps/Code to Reproduce

```
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn import datasets

iris = datasets.load_iris()
params = {'n_estimators':[10,20,30,40,50,60],
          'max_depth':[20,50,60,70,80]}
grid_obj = GridSearchCV(estimator=RandomForestClassifier(), param_grid=params, n_jobs=-1, verbose=1, cv=5)
grid_obj.fit(iris.data, iris.target)
```

### Expected Results

This is the output when using version 0.22.1
![image](https://user-images.githubusercontent.com/17684093/158401635-31755363-2108-41e7-8f7f-08985abd03b8.png)


### Actual Results

```Fitting 5 folds for each of 30 candidates, totalling 150 fits```

### Versions

```shell
System:
    python: 3.8.5 (default, Sep  3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)]
executable: D:\Programs\ApplicationsSetup\anaconda3\python.exe
   machine: Windows-10-10.0.19041-SP0

Python dependencies:
          pip: 21.2.2
   setuptools: 58.0.4
      sklearn: 1.0.2
        numpy: 1.19.2
        scipy: 1.6.2
       Cython: 0.29.25
       pandas: 1.4.1
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 2.2.0


!jupyter --version
Selected Jupyter core packages...
IPython          : 7.31.1
ipykernel        : 6.4.1
ipywidgets       : 7.6.5
jupyter_client   : 6.1.12
jupyter_core     : 4.9.1
jupyter_server   : 1.13.5
jupyterlab       : 3.2.9
nbclient         : 0.5.11
nbconvert        : 6.1.0
nbformat         : 5.1.3
notebook         : 6.4.9
qtconsole        : 5.2.2
traitlets        : 5.1.1
```

user: ahmadelsayed2009
labels: Needs Investigation: Issue requires investigation
issue_22836: state: open
title: Formula for dual gap of elastic net in coordinate descent solver
body: ### Describe the bug
The computation of the dual gap for the elastic net in the coordinate descent solver (`enet_coordinate_descent`) might be wrong.

The elastic net minimizes
```
Primal(w) = (1/2) * ||y - X w||_2^2 + alpha * ||w||_1 + beta/2 * ||w||_2^2
```

### Lasso
For the pure Lasso, i.e. `beta=0`, the dual becomes [1]
```
Dual(nu) = -1/2 * ||nu||_2^2 - nu'y   if ||X'nu||_∞ = max(abs(X'nu)) <= alpha   else -∞
```
with `nu = Xw - y` (=minus the residual `R`) as possible dual point. This yields the Lasso dual gap
```
Primal(w) - Dual(nu)
```
In case of `||X'nu||_∞ > alpha`, one uses a down-scaled variable `nu` in the dual.

### Elastic Net
For the elastic net, the dual, see section 5.2.3 in [2], becomes
```
Dual(nu) = -1/2 * ||nu||_2^2 - nu'y - 1/(2*beta) * sum_j (|X_j'nu| - alpha)_+^2
```
with `X_j` the j-th column of `X` and `x_+ = max(0, x)` the positive part.

### References
[1] Kim, Seung-Jean et al. “An Interior-Point Method for Large-Scale $\ell_1$-Regularized Least Squares.” IEEE Journal of Selected Topics in Signal Processing 1 (2007): 606-617. [pdf link](http://stanford.edu/~boyd/papers/pdf/l1_ls.pdf)
[2] Mendler-Dünner, Celestine et al. “Primal-Dual Rates and Certificates.” ICML (2016). [arxiv link](https://arxiv.org/pdf/1602.05205.pdf)

### Expected Results

```python
if beta > =0:
    R = y - X @ w  # note: R = -nu
    R_norm2 = R @ R
    gap = R_norm2 - R @ y + alpha * l1_norm + beta/2 * l2_norm
    gap += 1/(2*beta) * np.sum(np.max(0, np.abs(X.T @ R) - alpha)**2)
```
At least a test for it, similar to the pure Lasso case in https://github.com/scikit-learn/scikit-learn/blob/2c2f31d68c21b7647557b2f776e86c05954d80bf/sklearn/linear_model/tests/test_coordinate_descent.py#L292

### Actual Results

I start to be rattled by the `-beta * w` term in line 205.
https://github.com/scikit-learn/scikit-learn/blob/2c2f31d68c21b7647557b2f776e86c05954d80bf/sklearn/linear_model/_cd_fast.pyx#L205-L236



user: lorentzenchr
issue_22828: state: open
title: Gridsearch on TransformedTargetRegressor on pairwise data
body: Reported by @jajupmochi on gitter

When working with precomputed distances/kernels, the user might still want to perform certain transformations on the input or the output in a grid searched pipeline. At the moment, this code runs fine:

```py
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.neighbors import KNeighborsRegressor
from sklearn.model_selection import GridSearchCV

estimator = Pipeline([("model", KNeighborsRegressor(metric="precomputed"))])

grid_params = {"model__n_neighbors": [3, 5, 7]}
scoring = "neg_root_mean_squared_error"
clf = GridSearchCV(
    estimator,
    param_grid=grid_params,
    scoring=scoring,
    cv=5,
    return_train_score=True,
    refit=True,
    error_score="raise",
)

D_app = np.random.rand(10, 10)
y_app = np.random.rand(10)
clf.fit(D_app, y_app)
y_pred_app = clf.predict(D_app)
```

And it works fine because `_safe_split` (used in `GridSearchCV`) checks if the estimator is expecting pairwise data by querying the `pairwise` estimator tag and splits the data accordingly. And in the `Pipeline`, we check if the first step expects pairwise data:

https://github.com/scikit-learn/scikit-learn/blob/d3429ca328f37a6988bbb041b47ceeac4879d264/sklearn/pipeline.py#L706-L708

Now if the user uses a pipeline where the first step doesn't report requiring pairwise data, the error message becomes a lot harder to decrypt:

```py
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import MinMaxScaler
from sklearn.compose import TransformedTargetRegressor
from sklearn.neighbors import KNeighborsRegressor
from sklearn.model_selection import GridSearchCV

target_scaler = MinMaxScaler()
estimator = Pipeline(
    [
        (
            "model",
            TransformedTargetRegressor(
                KNeighborsRegressor(metric="precomputed"), transformer=target_scaler
            ),
        )
    ]
)

grid_params = {"model__regressor__n_neighbors": [3, 5, 7]}
scoring = "accuracy"
clf = GridSearchCV(
    estimator,
    param_grid=grid_params,
    scoring=scoring,
    cv=5,
    return_train_score=True,
    refit=True,
    error_score="raise",
)

D_app = np.random.rand(10, 10)
y_app = np.random.rand(10)
clf.fit(D_app, y_app)
```

which then results in the following error/trace:

```python-traceback
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/neighbors/_base.py", line 501, in _fit
    raise ValueError(
ValueError: Precomputed matrix must be square. Input is a 8x10 matrix.
```

Complete traceback:

<details>

``` python-traceback
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/adrin/.vscode-oss/extensions/ms-python.python-2022.2.1924087327/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/home/adrin/.vscode-oss/extensions/ms-python.python-2022.2.1924087327/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/home/adrin/.vscode-oss/extensions/ms-python.python-2022.2.1924087327/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/usr/lib/python3.10/runpy.py", line 269, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/tmp/2.py", line 24, in <module>
    clf.fit(D_app, y_app)
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/model_selection/_search.py", line 875, in fit
    self._run_search(evaluate_candidates)
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/model_selection/_search.py", line 1375, in _run_search
    evaluate_candidates(ParameterGrid(self.param_grid))
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/model_selection/_search.py", line 822, in evaluate_candidates
    out = parallel(
  File "/home/adrin/Projects/sklearn/.venv/lib/python3.10/site-packages/joblib/parallel.py", line 1043, in __call__
    if self.dispatch_one_batch(iterator):
  File "/home/adrin/Projects/sklearn/.venv/lib/python3.10/site-packages/joblib/parallel.py", line 861, in dispatch_one_batch
    self._dispatch(tasks)
  File "/home/adrin/Projects/sklearn/.venv/lib/python3.10/site-packages/joblib/parallel.py", line 779, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/home/adrin/Projects/sklearn/.venv/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 208, in apply_async
    result = ImmediateResult(func)
  File "/home/adrin/Projects/sklearn/.venv/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 572, in __init__
    self.results = batch()
  File "/home/adrin/Projects/sklearn/.venv/lib/python3.10/site-packages/joblib/parallel.py", line 262, in __call__
    return [func(*args, **kwargs)
  File "/home/adrin/Projects/sklearn/.venv/lib/python3.10/site-packages/joblib/parallel.py", line 262, in <listcomp>
    return [func(*args, **kwargs)
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/utils/fixes.py", line 118, in __call__
    return self.function(*args, **kwargs)
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/model_selection/_validation.py", line 687, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/pipeline.py", line 382, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/compose/_target.py", line 252, in fit
    self.regressor_.fit(X, y_trans, **fit_params)
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/neighbors/_regression.py", line 203, in fit
    return self._fit(X, y)
  File "/home/adrin/Projects/sklearn/scikit-learn/sklearn/neighbors/_base.py", line 501, in _fit
    raise ValueError(
ValueError: Precomputed matrix must be square. Input is a 8x10 matrix.
```

</details>

Now there are two issues here:

- `TransformedTargetRegressor` should probably work with pairwise data (having a quick look at the code I didn't find any place where we rely on `X` not being pairwise). This is probably an easy fix, we'd need to change `_more_tags` to:

``` py
    def _more_tags(self):
        regressor = self.regressor
        if regressor is None:
            from ..linear_model import LinearRegression

            regressor = LinearRegression()

        return {
            "pairwise": _safe_tags(self.regressor, "pairwise"),
            "poor_score": True,
            "multioutput": _safe_tags(regressor, key="multioutput"),
        }
```

- another potential fix is to catch the exception in `*GridSearch*` (and maybe `cross_val*` and raise a more meaningful error. I'm not sure how exactly though. 
user: adrinjalali
issue_22827: state: open
title: Improve tests by using global_random_seed fixture to make them less seed-sensitive
body: ## Context: the new `global_random_seed` fixture

#22749 introduces a new `global_random_seed` fixture to make it possible to run the same test with any seed between 0 and 99 included. By default, when `SKLEARN_TESTS_GLOBAL_RANDOM_SEED` is not set, this fixture is deterministically returning 42 to keep test runs deterministic by default and avoid any unnecessary disruption. However different CI builds set this seed to other arbitrary values (still deterministic) and nightly schedule builds on Azure now use `SKLEARN_TESTS_GLOBAL_RANDOM_SEED="any"` to progressively explore any seed on the 0-99 range.

## Motivation

The aim of this new fixture is to make sure that we avoid writing tests that artificially depend on a specific value of the random seed and therefore hiding a real mathematical problem in our code unknowingly (see e.g. https://github.com/scikit-learn/scikit-learn/pull/21701#discussion_r823847947). At the same time we still want to keep the test deterministic and independent of the execution order by default to avoid introducing unnecessary maintenance overhead.

In addition to making the tests insensitive, randomizing those tests with different seeds has the side benefit of making the assertions of those tests robust to small numerical variations that could otherwise stem from other sources such as platform-specific / dependency-specific numerical rounding variations that we do not cover in our existing CI infrastructure.

More details about the fixture in the online dev doc for the `SKLEARN_TESTS_GLOBAL_RANDOM_SEED` env variable:

https://scikit-learn.org/dev/computing/parallelism.html#environment-variables

## Guidelines to convert existing tests

- We probably do not need to convert all scikit-learn tests to use this fixture. We should instead focus our efforts on tests that actually check for **important mathematical properties** of our estimators or model evaluation tools. For instance, there is no need to check for the seed-insensitivity of tests that checks for the exception messages raised when passing invalid inputs.

- To avoid having to review huge PRs that impact many files at once and can lead to conflicts, let's open PRs that edit at most one test file at a time. For instance use a title such as:

> TST use global_random_seed in sklearn/_loss/tests/test_glm_distribution.py

- Please reference `#22827` in the description of the PR and put the full filename of the test file you edit in the title of the PR.

- To convert an existing test with a fixed seed, the general pattern is to rewrite a function such as:

```python
def test_some_function():
    rng = np.random.RandomState(0)
    ...
```

to:

```python
def test_some_function(global_random_seed):
    rng = np.random.RandomState(global_random_seed)
    ...
```

and then check that the test function is actually seed-insensitive by running with all seeds between 0 and 99 locally (can be slow! only run for one specific test at a time!):

```
SKLEARN_TESTS_GLOBAL_RANDOM_SEED="all" pytest sklearn/some_module/test/test_some_module.py -k test_some_function
```

If this is not the case, the test will probably need to be reworked to find a more stable to way to check the interesting mathematical properties.

- if the failing assertions are related to the generalization performance of a model, maybe the training set size should be slightly bigger (while keeping the test runtime as fast as possible), or with fewer noisy features or the training should be done with stronger regularization. Or more simply we can relax the tolerance threshold while ensuring it does not become trivial (e.g. by comparing to a trivial baseline);

- if the failing assertions depend on some regularities of a synthetically generated dataset, making decreasing the noise level of the datasets;

- some tests might also fail when encountering data that trigger edge cases such as (near-)tied distances between datapoints that make the outcome of computation unstable. Changing the data generation code to significantly decrease the likelihood of those edge case (e.g. by adding more noise to the input features) can help in those cases.

- **Note**: in most cases, tweaking the tolerances of the assertions is **not** the appropriate way to make the tests pass. The first thing to do is try to understand what the test is checking, if the test is correct, if the expectations of the test are realistic. Then if the test seems correct and *should* pass for all random seed but doesn't, investigate if the estimator or function is bugged. As a last resort, tolerances can be loosened if the test is considered valid but aims to check a statistical property that is highly sensitive to the random seed.

In some cases, it might be very hard to write a seed-insensitive test that tolerate all seeds between 0 and 99 while still running in less than 1s. In those (hopefully rare) cases, I think it's fine to reduce the range of admissible seeds with the following pattern:

```python
def test_some_function(global_random_seed):
    # Making this test seed-insensitive for the 0-99 range would
    # be too costly. Restricting to the 0-9 range is necessary to
    # use small enough datasets that avoid increasing the run time
    # too much.
    rng = np.random.RandomState(global_random_seed % 10)
    ...
```

- Run the CI for tests that take a `global_random_seed` by pushing a commit message with the following structure:

```
<title> [all random seeds]
<test_name_1>
<test_name_2>
...
```

Note, running `git commit --allow-empty` allows you to have a commit message without any changes.

## List of test modules to upgrade

```
find sklearn -name "test_*.py"
```

- [x] sklearn/_loss/tests/test_glm_distribution.py
- [x] sklearn/_loss/tests/test_link.py
- [x] sklearn/_loss/tests/test_loss.py  #22847
- [x] sklearn/cluster/tests/test_affinity_propagation.py
- [x] sklearn/cluster/tests/test_bicluster.py
- [x] sklearn/cluster/tests/test_birch.py
- [x] sklearn/cluster/tests/test_dbscan.py
- [x] sklearn/cluster/tests/test_feature_agglomeration.py https://github.com/scikit-learn/scikit-learn/pull/23700
- [x] sklearn/cluster/tests/test_hierarchical.py
- [x] sklearn/cluster/tests/test_k_means.py
- [ ] sklearn/cluster/tests/test_mean_shift.py
- [ ] sklearn/cluster/tests/test_optics.py
- [ ] sklearn/cluster/tests/test_spectral.py
- [ ] sklearn/compose/tests/test_column_transformer.py
- [ ] sklearn/compose/tests/test_target.py
- [x] sklearn/covariance/tests/test_covariance.py
- [x] sklearn/covariance/tests/test_elliptic_envelope.py
- [ ] sklearn/covariance/tests/test_graphical_lasso.py
- [x] sklearn/covariance/tests/test_robust_covariance.py
- [x] sklearn/cross_decomposition/tests/test_pls.py
- [x] sklearn/datasets/tests/test_20news.py
- [x] sklearn/datasets/tests/test_base.py
- [x] sklearn/datasets/tests/test_california_housing.py
- [x] sklearn/datasets/tests/test_common.py
- [x] sklearn/datasets/tests/test_covtype.py
- [x] sklearn/datasets/tests/test_kddcup99.py
- [x] sklearn/datasets/tests/test_lfw.py
- [x] sklearn/datasets/tests/test_olivetti_faces.py
- [x] sklearn/datasets/tests/test_openml.py
- [x] sklearn/datasets/tests/test_rcv1.py
- [ ] sklearn/datasets/tests/test_samples_generator.py
- [ ] sklearn/datasets/tests/test_svmlight_format.py
- [ ] sklearn/decomposition/tests/test_dict_learning.py
- [ ] sklearn/decomposition/tests/test_factor_analysis.py
- [ ] sklearn/decomposition/tests/test_fastica.py
- [ ] sklearn/decomposition/tests/test_incremental_pca.py
- [ ] sklearn/decomposition/tests/test_kernel_pca.py
- [ ] sklearn/decomposition/tests/test_nmf.py
- [ ] sklearn/decomposition/tests/test_online_lda.py
- [ ] sklearn/decomposition/tests/test_pca.py
- [ ] sklearn/decomposition/tests/test_sparse_pca.py
- [ ] sklearn/decomposition/tests/test_truncated_svd.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_binning.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_bitset.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_compare_lightgbm.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_grower.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_histogram.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_monotonic_contraints.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_predictor.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_splitting.py
- [ ] sklearn/ensemble/_hist_gradient_boosting/tests/test_warm_start.py
- [ ] sklearn/ensemble/tests/test_bagging.py
- [x] sklearn/ensemble/tests/test_base.py
- [ ] sklearn/ensemble/tests/test_common.py
- [ ] sklearn/ensemble/tests/test_forest.py
- [x] sklearn/ensemble/tests/test_gradient_boosting_loss_functions.py
- [x] sklearn/ensemble/tests/test_gradient_boosting.py
- [x] sklearn/ensemble/tests/test_iforest.py #22901
- [ ] sklearn/ensemble/tests/test_stacking.py
- [x] sklearn/ensemble/tests/test_voting.py
- [ ] sklearn/ensemble/tests/test_weight_boosting.py
- [ ] sklearn/experimental/tests/test_enable_hist_gradient_boosting.py
- [ ] sklearn/experimental/tests/test_enable_iterative_imputer.py
- [ ] sklearn/experimental/tests/test_enable_successive_halving.py
- [x] sklearn/feature_extraction/tests/test_dict_vectorizer.py
- [ ] sklearn/feature_extraction/tests/test_feature_hasher.py
- [ ] sklearn/feature_extraction/tests/test_image.py
- [ ] sklearn/feature_extraction/tests/test_text.py
- [ ] sklearn/feature_selection/tests/test_base.py
- [ ] sklearn/feature_selection/tests/test_chi2.py
- [ ] sklearn/feature_selection/tests/test_feature_select.py
- [ ] sklearn/feature_selection/tests/test_from_model.py
- [ ] sklearn/feature_selection/tests/test_mutual_info.py
- [x] sklearn/feature_selection/tests/test_rfe.py
- [ ] sklearn/feature_selection/tests/test_sequential.py
- [ ] sklearn/feature_selection/tests/test_variance_threshold.py
- [x] sklearn/gaussian_process/tests/test_gpc.py
- [ ] sklearn/gaussian_process/tests/test_gpr.py
- [ ] sklearn/gaussian_process/tests/test_kernels.py
- [ ] sklearn/impute/tests/test_base.py
- [ ] sklearn/impute/tests/test_common.py
- [ ] sklearn/impute/tests/test_impute.py
- [ ] sklearn/impute/tests/test_knn.py
- [ ] sklearn/inspection/_plot/tests/test_plot_partial_dependence.py
- [ ] sklearn/inspection/tests/test_partial_dependence.py
- [ ] sklearn/inspection/tests/test_permutation_importance.py
- [ ] sklearn/linear_model/_glm/tests/test_glm.py
- [ ] sklearn/linear_model/_glm/tests/test_link.py
- [x] sklearn/linear_model/tests/test_base.py
- [ ] sklearn/linear_model/tests/test_bayes.py
- [ ] sklearn/linear_model/tests/test_common.py
- [ ] sklearn/linear_model/tests/test_coordinate_descent.py
- [ ] sklearn/linear_model/tests/test_huber.py
- [ ] sklearn/linear_model/tests/test_least_angle.py
- [ ] sklearn/linear_model/tests/test_linear_loss.py
- [ ] sklearn/linear_model/tests/test_logistic.py
- [ ] sklearn/linear_model/tests/test_omp.py
- [ ] sklearn/linear_model/tests/test_passive_aggressive.py
- [ ] sklearn/linear_model/tests/test_perceptron.py
- [ ] sklearn/linear_model/tests/test_quantile.py
- [ ] sklearn/linear_model/tests/test_ransac.py
- [ ] sklearn/linear_model/tests/test_ridge.py
- [ ] sklearn/linear_model/tests/test_sag.py
- [ ] sklearn/linear_model/tests/test_sgd.py
- [ ] sklearn/linear_model/tests/test_sparse_coordinate_descent.py
- [ ] sklearn/linear_model/tests/test_theil_sen.py
- [ ] sklearn/manifold/tests/test_isomap.py
- [ ] sklearn/manifold/tests/test_locally_linear.py
- [ ] sklearn/manifold/tests/test_mds.py
- [ ] sklearn/manifold/tests/test_spectral_embedding.py
- [ ] sklearn/manifold/tests/test_t_sne.py
- [ ] sklearn/metrics/_plot/tests/test_base.py
- [ ] sklearn/metrics/_plot/tests/test_common_curve_display.py
- [ ] sklearn/metrics/_plot/tests/test_confusion_matrix_display.py
- [ ] sklearn/metrics/_plot/tests/test_det_curve_display.py
- [ ] sklearn/metrics/_plot/tests/test_plot_confusion_matrix.py
- [ ] sklearn/metrics/_plot/tests/test_plot_curve_common.py
- [ ] sklearn/metrics/_plot/tests/test_plot_det_curve.py
- [ ] sklearn/metrics/_plot/tests/test_plot_precision_recall.py
- [ ] sklearn/metrics/_plot/tests/test_plot_roc_curve.py
- [ ] sklearn/metrics/_plot/tests/test_precision_recall_display.py
- [ ] sklearn/metrics/_plot/tests/test_roc_curve_display.py
- [ ] sklearn/metrics/cluster/tests/test_bicluster.py
- [ ] sklearn/metrics/cluster/tests/test_common.py
- [ ] sklearn/metrics/cluster/tests/test_supervised.py
- [ ] sklearn/metrics/cluster/tests/test_unsupervised.py
- [ ] sklearn/metrics/tests/test_classification.py
- [ ] sklearn/metrics/tests/test_common.py
- [ ] sklearn/metrics/tests/test_dist_metrics.py
- [x] sklearn/metrics/tests/test_pairwise_distances_reduction.py https://github.com/scikit-learn/scikit-learn/pull/22862
- [ ] sklearn/metrics/tests/test_pairwise.py
- [ ] sklearn/metrics/tests/test_ranking.py
- [ ] sklearn/metrics/tests/test_regression.py
- [ ] sklearn/metrics/tests/test_score_objects.py
- [ ] sklearn/mixture/tests/test_bayesian_mixture.py
- [ ] sklearn/mixture/tests/test_gaussian_mixture.py
- [x] sklearn/mixture/tests/test_mixture.py
- [ ] sklearn/model_selection/tests/test_search.py
- [ ] sklearn/model_selection/tests/test_split.py
- [ ] sklearn/model_selection/tests/test_successive_halving.py
- [ ] sklearn/model_selection/tests/test_validation.py
- [ ] sklearn/neighbors/tests/test_ball_tree.py
- [ ] sklearn/neighbors/tests/test_graph.py
- [ ] sklearn/neighbors/tests/test_kd_tree.py
- [ ] sklearn/neighbors/tests/test_kde.py
- [ ] sklearn/neighbors/tests/test_lof.py
- [ ] sklearn/neighbors/tests/test_nca.py
- [ ] sklearn/neighbors/tests/test_nearest_centroid.py
- [ ] sklearn/neighbors/tests/test_neighbors_pipeline.py
- [ ] sklearn/neighbors/tests/test_neighbors_tree.py
- [ ] sklearn/neighbors/tests/test_neighbors.py
- [ ] sklearn/neighbors/tests/test_quad_tree.py
- [ ] sklearn/neural_network/tests/test_base.py
- [ ] sklearn/neural_network/tests/test_mlp.py
- [ ] sklearn/neural_network/tests/test_rbm.py
- [ ] sklearn/neural_network/tests/test_stochastic_optimizers.py
- [ ] sklearn/preprocessing/tests/test_common.py
- [ ] sklearn/preprocessing/tests/test_data.py
- [ ] sklearn/preprocessing/tests/test_discretization.py
- [ ] sklearn/preprocessing/tests/test_encoders.py
- [ ] sklearn/preprocessing/tests/test_function_transformer.py
- [ ] sklearn/preprocessing/tests/test_label.py
- [ ] sklearn/preprocessing/tests/test_polynomial.py
- [ ] sklearn/semi_supervised/tests/test_label_propagation.py
- [ ] sklearn/semi_supervised/tests/test_self_training.py
- [ ] sklearn/svm/tests/test_bounds.py
- [ ] sklearn/svm/tests/test_sparse.py
- [ ] sklearn/svm/tests/test_svm.py
- [ ] sklearn/tests/test_base.py
- [ ] sklearn/tests/test_build.py
- [ ] sklearn/tests/test_calibration.py
- [ ] sklearn/tests/test_check_build.py
- [ ] sklearn/tests/test_common.py
- [ ] sklearn/tests/test_config.py
- [ ] sklearn/tests/test_discriminant_analysis.py
- [ ] sklearn/tests/test_docstring_parameters.py
- [ ] sklearn/tests/test_docstrings.py
- [x] sklearn/tests/test_dummy.py
- [ ] sklearn/tests/test_init.py
- [ ] sklearn/tests/test_isotonic.py
- [ ] sklearn/tests/test_kernel_approximation.py
- [ ] sklearn/tests/test_kernel_ridge.py
- [ ] sklearn/tests/test_metaestimators.py
- [ ] sklearn/tests/test_min_dependencies_readme.py
- [ ] sklearn/tests/test_multiclass.py
- [ ] sklearn/tests/test_multioutput.py
- [ ] sklearn/tests/test_naive_bayes.py
- [ ] sklearn/tests/test_pipeline.py
- [ ] sklearn/tests/test_random_projection.py
- [ ] sklearn/tree/tests/test_export.py
- [ ] sklearn/tree/tests/test_reingold_tilford.py
- [ ] sklearn/tree/tests/test_tree.py
- [ ] sklearn/utils/tests/test_arpack.py
- [ ] sklearn/utils/tests/test_arrayfuncs.py
- [ ] sklearn/utils/tests/test_class_weight.py
- [ ] sklearn/utils/tests/test_cython_blas.py
- [ ] sklearn/utils/tests/test_cython_templating.py
- [ ] sklearn/utils/tests/test_deprecation.py
- [ ] sklearn/utils/tests/test_encode.py
- [ ] sklearn/utils/tests/test_estimator_checks.py
- [ ] sklearn/utils/tests/test_estimator_html_repr.py
- [ ] sklearn/utils/tests/test_extmath.py
- [ ] sklearn/utils/tests/test_fast_dict.py
- [ ] sklearn/utils/tests/test_fixes.py
- [ ] sklearn/utils/tests/test_graph.py
- [ ] sklearn/utils/tests/test_metaestimators.py
- [ ] sklearn/utils/tests/test_mocking.py
- [ ] sklearn/utils/tests/test_multiclass.py
- [ ] sklearn/utils/tests/test_murmurhash.py
- [ ] sklearn/utils/tests/test_optimize.py
- [ ] sklearn/utils/tests/test_parallel.py
- [ ] sklearn/utils/tests/test_pprint.py
- [ ] sklearn/utils/tests/test_random.py
- [ ] sklearn/utils/tests/test_readonly_wrapper.py
- [ ] sklearn/utils/tests/test_seq_dataset.py
- [ ] sklearn/utils/tests/test_shortest_path.py
- [ ] sklearn/utils/tests/test_show_versions.py
- [ ] sklearn/utils/tests/test_sparsefuncs.py
- [ ] sklearn/utils/tests/test_stats.py
- [ ] sklearn/utils/tests/test_tags.py
- [ ] sklearn/utils/tests/test_testing.py
- [ ] sklearn/utils/tests/test_utils.py
- [ ] sklearn/utils/tests/test_validation.py
- [ ] sklearn/utils/tests/test_weight_vector.py

Note that some of those files might not have any test to update.

user: ogrisel
labels: Hard: Hard level of difficulty
module:test-suite: everything related to our tests
Meta-issue: General issue associated to an identified list of tasks
issue_22818: state: open
title: Change how KNeighborsClassifier handles tiebreakers
body: ### Describe the workflow you want to enable

I see that KNeighborsClassifier handles tiebreakers by taking the mode of the classes contained with the tie. What about using the sum of the distances as the tiebreaker instead (of using the mode.)

I feel that taking the least sum of distances is a better educated guess with minimal changes to the code.

### Describe your proposed solution

Create two dictionaries to keep a count of the Class count and the sum of Class distances.
If there is no tie return the Class value that won the vote (as normal).
If there is a tie return the Class that has the minimum sum of distances (instead of what scipy.stats.mode yields which I believe will always be the smallest Class value from the Classes within the tie.)

```
    dict_count = {}
    dict_dist = {}
    
    for i in range(num_neighbors):
        
        # Get the class from the i-th neighbor - here my Class value is at offset 13
        c = neighbors[i][13]
        
        # Keep a dictionary of Class counts
        if c in dict_count:
            dict_count[c] = dict_count[c] + 1
        else:
            dict_count[c] = 1
          
     # Keep a dictionary of total Class distances 
        if c in dict_dist:
            dict_dist[c] = dict_dist[c] + distances[i]
        else:
            dict_dist[c] = distances[i]
    
    # Return the key within the dictionary that has the largest value
    best_keys = [key for key, value in dict_count.items() if value == max(dict_count.values())]

    if len(best_keys) == 1:
        prediction = best_keys[0]

    else:   # There was a tie
        tie_dict = {}
        for b in best_keys:
            tie_dict[b] = dict_dist[b]
            
        prediction = min(tie_dict, key=tie_dict.get)

    return prediction
```

### Describe alternatives you've considered, if relevant

Writing my own _k_-NN algorithm 🙁

### Additional context

_No response_
user: raymondj-pace
labels: Needs Investigation: Issue requires investigation
issue_22770: state: open
title: Allow other initialization strategies for IterativeImputer
body: ### Describe the workflow you want to enable

I want to initialize an `IterativeImputer` using a `KNNImputer` rather than a `SimpleImputer`.

### Describe your proposed solution

Modifying `IterativeImputer` to accept instances of `_BaseImputer` as `initial_strategy`s should be fairly simple, though I'm aware it is under active development and I understand why this wouldn't be a priority. I'm happy to put together a pull request if it is likely to be accepted.

### Describe alternatives you've considered, if relevant

I am not aware of any compelling alternatives. 

### Additional context

_No response_
user: eschibli
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_22754: state: open
title: FEA Add Oblique trees and oblique splitters to tree module: Enables extensibility of the trees
body: #### Reference Issues/PRs
Closes: #20819 

This is ready for review.

#### What does this implement/fix? Explain your changes.

Implements oblique trees/splitter as a subclass of the existing `Tree`/`Splitter`. This further allows extensibility of the sklearn `Tree` and `Splitter` to allow downstream packages that have to define more complex and exotic sampling mechanisms. The total Cython code that constitutes the logic addition is around ~950 LOC, while the rest are from unit tests, and adding the Python API.

This is an extension of decision trees to generalize to random linear combination of feature oblique trees proposed by Breimann in 2001. This will enable 3rd party packages to subclass this code to instantiate other trees that work on "combinations" of features in some way (e.g. taking the weighted sum, or a kernel). The general intuition is that OF can sample more diverse set of features enabling better generalization and robustness to high-dimensional noise. RF should be used if the user suspects all data is aligned on the feature dimensions. The tradeoffs of computation time, model size and scoring time are complex because OF can fit shorter trees, but requires storage of additional data. OF needs to perform additional operations increasing computation time, but it can be negligible. We always suggest using OF first if you have the computational and storage flexibility. We suggest using RF for now if you have very large sample sizes and very very strict runtime/storage constraints.

- **Inclusion criterion** was discussed. Breiman 2001, proposed oblique forests and demonstrated its superiority. Also see https://github.com/scikit-learn/scikit-learn/issues/20819#issuecomment-1065523071
- **Maintainability** was addressed by figuring out a way that `ObliqueSplitter` and `ObliqueTree` can subclass the existing code. 
- **Runtime performance** is benchmarked (https://gist.github.com/adam2392/a652aa1c88ba94f0d6aab96ccd3ade24)
- **Interpretability** is demonstrated using an extension of the RandomForest example with `iris` dataset (https://gist.github.com/adam2392/78519091104cecfeb8ff796eac7e8115)

##### Experiments supporting the change

Requested by Thomas Fan, Guillaume, and Olivier, we ran the following experiments.

- runtime: see above
- visual interpretation of the decision boundaries: see iris example
- grid search outcomes on simulated data of OF vs RF: https://nbviewer.org/github/neurodata/scikit-learn/blob/5e8f1a6a506bf77fc99cdb59d81fe02f8a301e20/notebook/simulation_benchmark_OF_vs_RF.ipynb
- grid search outcomes on low-dimension MNIST of OF vs RF: https://nbviewer.org/github/neurodata/scikit-learn/blob/25a1e9819c5def42f4f919eafdf4df4b065b2c08/notebook/mnist_benchmark_OF_vs_RF.ipynb
- performance (accuracy) vs score time: see above 2 notebooks
- performance vs fit time: see above 2 notebooks
- performance vs model size: see above 2 notebooks
- grid search outcomes on an openml dataset: TBD

##### Docs changes for education and user-guidance

- [x] extension of the `examples/tree/plot_iris_dtc.py` to include oblique trees
- [x] `modules/tree.rst` with a new section on oblique trees
- [x] `modules/ensemble.rst` with a new section on oblique forests
- [ ] A new example in `examples/ensemble/plot_oblique_axis_aligned_forests.py` demonstrating oblique forest superiority on a real dataset from openml and a simulated dataset

The changes to `setup.py` files were necessary to compile package in some of the CI pipelines. It worked for me w/o it locally on my Mac M1 machine though.

#### Tracking Progress on sub-items - used for keeping track of challenges that arose (optional read)
The remaining items to complete are:
- [x] Add unit tests
- [x] Refactor to use memory views as in normal splitter (https://github.com/scikit-learn/scikit-learn/pull/23273/files)
- [x] Fix pickling issue in https://github.com/scikit-learn/scikit-learn/pull/22754#issuecomment-1128124308
- [x] Add `feature_importances`, or specify an error message if it's not allowed.
- [x] Add extension to the RF-Iris dataset example done by Jong Shin within the docs. Also adding a synthetic dataset perhaps that looks cleaner. And another dataset, for example `digits` (8 x 8 images) that can get shallower/smaller-forests vs RandomForest; e.g. train OF/RF with 100 trees, and subsample the forest and plot accuracy and plot the max_depth.  Basically get a few more examples that are candidates to include into the docs: 1 example showing an intuitive visualization and then another example to demonstrate possible superiority in terms of real example. E.g. digits horizontally stack the real signal but then create permuted noisy copies and stack again. Put it into a separate notebook for now for downstream integration into scikit-learn. End goal: education for users on the inductive bias for oblique trees and when to use.
- [x] Determine if minor refactoring is needed with `TreeBuilder/Tree` for adding nodes (see: https://github.com/scikit-learn/scikit-learn/pull/22756#issuecomment-1065549821)

#### Any other comments?
This is the PR to demonstrate the end product. To simplify the merging process, I propose the merging of this PR in a series of other smaller PRs.

- [x] ~#22753~
- [x] Fix unit test organization for pickling trees: https://github.com/scikit-learn/scikit-learn/pull/22915#pullrequestreview-918823308
- [x] Small PR to update `Tree` abstractions, such as in `sklearn/tree/_classes.py` and `sklearn/tree/_tree.pyx`
- [x] Simplifications of the utility code, such as sorting to use c++ (https://github.com/scikit-learn/scikit-learn/pull/22760)

Another misc. note is that oblique and decision trees can be further easily improved in terms of training speed via quantization of continuous feature values in the data (i.e. same idea used in histogram gradient boosting).


user: adam2392
issue_22733: state: open
title: [MRG] Set the target of datasets to be pandas categoricals
body: Currently the `datasets` functions `load_wine`, `load_iris`, `load_breast_cancer` and `load_digits` have the option of returning their data as a pandas `DataFrame`. In this case, the data returned as the `target` has the data type `int`:

```python
from sklearn.datasets import load_wine
wine = load_wines(as_frame=True)
assert wine.target.dtype == "int64"
```

This PR changes this so that the target is set to be a [pandas categorical](https://pandas.pydata.org/docs/user_guide/categorical.html) so that now:

```python
assert wine.target.dtype == pd.CategoricalDtype(categories=[0, 1, 2])
```

This allows a number of niceties. Firstly, those who are used to using Pandas will expect categories to be `CategoricalDtype` as it is conceptually different to an integer. Secondly, it allows the option of:

```python
y = wine.target.cat.rename_categories(wine.target_names)
```

which allows the rows to be labelled, making plotting and visualisation legends easier:

```
0         setosa
1         setosa
         ...    
176    virginica
177    virginica
Name: target, Length: 178, dtype: category
Categories (3, object): ['setosa', 'versicolor', 'virginica']
```

Finally, some tools (like seaborn) treat `int` dtype as a continuous variable when plotting and will set a continuous colour palette:
```python
sns.relplot(data=wine.data, x="alcohol", y="proline", hue=wine.target)
```
![int](https://user-images.githubusercontent.com/61316/157282513-8cf2f757-0bcf-470c-82b1-5f0d1951da50.png)
but if the target is a category then it looks like:
![cat](https://user-images.githubusercontent.com/61316/157282665-9e172934-dcfe-4069-887e-01bf348d97d1.png)

A category containing integers is represented in the underlying numpy array in the same way as before so scikit-learn will train on it the same.

In the future, functions like `sklearn.datasets.make_classification` could do the same, but at the moment they do not have the option of returning a `DataFrame`. We could also call the `rename_categories` function on the data for the user, but I'm not sure if all scikit-learn models will work with non-integer values for the target category?

user: milliams
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_22709: state: open
title: Create a similar class to KMeans that uses medians instead of means (KMedians)
body: ### Describe the workflow you want to enable

I would like a new class: sklearn.cluster.KMedians (or an option to sklearn.cluster.KMeans) that allows the methods to use medians instead of means.

K-n clustering can greatly improve some instances where there are outliers. Using the median minimizes outliers.

### Describe your proposed solution

Create a new class sklearn.cluster.KMedians that works the same as sklearn.cluster.KMeans but instead uses median to compute the new centroids (instead of using mean)

### Describe alternatives you've considered, if relevant

Using a version I wrote:

```
class mu_type(enum.IntEnum):
    mean = 1
    median = 2


#
# Euclidean distance between two vectors
#
def euclidean_distance(row1, row2):
    distance = 0.0
    for i in range(len(row1)):
        distance += (row1[i] - row2[i])**2
    return math.sqrt(distance)


#
# k-means and k-medians clustering
# mu defines if the algorithm runs as k-means or k-medians
#
def k_m_clustering_2(x, c, mu=mu_type.mean):
    
    d = [[0]*len(c) for i1 in range(len(x))]
    
    l = [0]*len(x)
    c_last = [0]*len(c)
    
    while c_last != c:
        
        # Save the last list of center points to compare for updates later
        c_last = c.copy()
        
        for i in range(len(x)):
            for j in range(len(c)):
                if DEBUG:
                    print('distance between: ', end='')
                    print(x[i], end='')
                    print(' and ', end='')
                    print(c[j], end='')
                    
                d[i][j] = euclidean_distance(x[i], c[j])
                
                if DEBUG:
                    print(' = ', end='')
                    print(d[i][j])
                    
            l[i] = d[i].index(min(d[i]))

        if DEBUG:
            print("L = ", end='')
            print(l)
            print()
            print()
        
        if DEBUG:
            print("Old center points: ", end='')
            print(c)
            print()

        #
        # Update center points
        #
        for i in range(len(c)):
    
            #
            # Compute mean or median for new center points
            #       
            if mu == mu_type.mean:
                count = 0
                _x = []
                _y = []
                for j in range(len(l)):
                    if l[j] == i:
                        count += 1
                        _x.append(x[j][0])
                        _y.append(x[j][1])
                c[i] = [(sum(_x)/count), (sum(_y)/count)]
                
            elif mu == mu_type.median:
                x_y = []
                for j in range(len(l)):
                    if l[j] == i:
                        x_y.append([x[j][0], x[j][1]])
                c[i] = median(x_y)
            
        if DEBUG:
            print("New center points: ", end='')
            print(c)
            print()
    
    if DEBUG:
        print("Center points have not changed\n")
        print("Final: ", end='')
        print(l)
        print()
    
    return c, l




# Call it:

DEBUG=True

c = [[2, 2], [3, 4], [6, 2]]
x = [[1, 2], [2, 1], [1, 3], [5, 4], [6, 3], [7, 2], [6, 1]]

c_arr, l_arr = k_m_clustering_2(x, c, mu_type.median)

for i in range(len(x)):
    print('x' + str(i+1) + ' = ' + str(l_arr[i]))
print('\n')
```

Output:

distance between: [1, 2] and [2, 2] = 1.0
distance between: [1, 2] and [3, 4] = 2.8284271247461903
distance between: [1, 2] and [6, 2] = 5.0
distance between: [2, 1] and [2, 2] = 1.0
distance between: [2, 1] and [3, 4] = 3.1622776601683795
distance between: [2, 1] and [6, 2] = 4.123105625617661
distance between: [1, 3] and [2, 2] = 1.4142135623730951
distance between: [1, 3] and [3, 4] = 2.23606797749979
distance between: [1, 3] and [6, 2] = 5.0990195135927845
distance between: [5, 4] and [2, 2] = 3.605551275463989
distance between: [5, 4] and [3, 4] = 2.0
distance between: [5, 4] and [6, 2] = 2.23606797749979
distance between: [6, 3] and [2, 2] = 4.123105625617661
distance between: [6, 3] and [3, 4] = 3.1622776601683795
distance between: [6, 3] and [6, 2] = 1.0
distance between: [7, 2] and [2, 2] = 5.0
distance between: [7, 2] and [3, 4] = 4.47213595499958
distance between: [7, 2] and [6, 2] = 1.0
distance between: [6, 1] and [2, 2] = 4.123105625617661
distance between: [6, 1] and [3, 4] = 4.242640687119285
distance between: [6, 1] and [6, 2] = 1.0
L = [0, 0, 0, 1, 2, 2, 2]


Old center points: [[2, 2], [3, 4], [6, 2]]

New center points: [[1, 3], [5, 4], [6, 3]]

distance between: [1, 2] and [1, 3] = 1.0
distance between: [1, 2] and [5, 4] = 4.47213595499958
distance between: [1, 2] and [6, 3] = 5.0990195135927845
distance between: [2, 1] and [1, 3] = 2.23606797749979
distance between: [2, 1] and [5, 4] = 4.242640687119285
distance between: [2, 1] and [6, 3] = 4.47213595499958
distance between: [1, 3] and [1, 3] = 0.0
distance between: [1, 3] and [5, 4] = 4.123105625617661
distance between: [1, 3] and [6, 3] = 5.0
distance between: [5, 4] and [1, 3] = 4.123105625617661
distance between: [5, 4] and [5, 4] = 0.0
distance between: [5, 4] and [6, 3] = 1.4142135623730951
distance between: [6, 3] and [1, 3] = 5.0
distance between: [6, 3] and [5, 4] = 1.4142135623730951
distance between: [6, 3] and [6, 3] = 0.0
distance between: [7, 2] and [1, 3] = 6.082762530298219
distance between: [7, 2] and [5, 4] = 2.8284271247461903
distance between: [7, 2] and [6, 3] = 1.4142135623730951
distance between: [6, 1] and [1, 3] = 5.385164807134504
distance between: [6, 1] and [5, 4] = 3.1622776601683795
distance between: [6, 1] and [6, 3] = 2.0
L = [0, 0, 0, 1, 2, 2, 2]


Old center points: [[1, 3], [5, 4], [6, 3]]

New center points: [[1, 3], [5, 4], [6, 3]]

Center points have not changed

Final: [0, 0, 0, 1, 2, 2, 2]

x1 = 0
x2 = 0
x3 = 0
x4 = 1
x5 = 2
x6 = 2
x7 = 2


### Additional context

_No response_
user: raymondj-pace
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_22708: state: open
title: Implement Repeated Group CV
body: https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/model_selection/_split.py#L505

I tried to implement repeated group cv using GroupKFold and _RepeatedSplits. But it did not work unless I included `shuffle=False, random_state=None` to `def __ini__ ()`

I just want to check if I am why `shuffle=False, random_state=None` got excluded from `def __ini__ ()`, and if it is okay for me to do the following implementation: `def __init__(self, n_splits=5, shuffle=False, random_state=None)`
user: weiweijin722
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_22705: state: open
title: ENH Paired haversine
body: Closes https://github.com/scikit-learn/scikit-learn/issues/17212

There is a PR in https://github.com/scikit-learn/scikit-learn/pull/17258/ , but it has grown stale.

(I also realised there was an issue in the original implementation, in that the existing haversine expects the input to be in radians.)

~WIP because I cannot build sklearn to test locally right now.~
user: jona-sassenhagen
labels: Needs Decision: Requires decision
Waiting for Second Reviewer: First reviewer is done, need a second one!
issue_22691: state: open
title: Include entire range in `check_scalar` error message
body: Currently docstrings description for scalar ranges uses the interval syntax:

https://github.com/scikit-learn/scikit-learn/blob/42cc05c5ddac0e0c4392871a6825c53ac88ace36/sklearn/linear_model/_glm/glm.py#L462

While the error message uses a different notation:

```python
from sklearn.utils.validation import check_scalar

check_scalar(-1.0, "alpha", target_type=float, min_val=0.0)
# ValueError: alpha == -1.0, must be >= 0.0.
```

Should we update this notation to be consistent with the docstring? i.e:

```python
# ValueError: alpha == -1.0, must be in the range [0.0, inf)
```

(I am assuming that `inf` is **always** an open boundary)

CC @glemaitre 
user: thomasjpfan
issue_22682: state: open
title: Estimator check for dtype preservation for regressors
body: ### Describe the workflow you want to enable

As discussed in https://github.com/scikit-learn/scikit-learn/pull/22663#issuecomment-1058368882, we should have a common test that checks that the `predict` method of regressors preserves the dtype, similarly to `check_transformer_preserve_dtypes` for transformers.

More specifically I would expect the following to hold most of the time:

```python
>>> dtype = np.float32
>>> reg = Regressor().fit(X_train.astype(dtype), y_train.astype(dtype))
>>> assert reg.predict(X_test.astype(dtype)).dtype == dtype
```

If `X_train`, `y_train`, and `X_test` do not all share the same dtype, I would be in favor of leaving the behavior undefined (and unchecked).

This could be coupled with the existing `preserves_dtype` estimator tag.



### Describe your proposed solution

- Implement a new `check_regressor_preserve_dtypes` function, next to `check_transformer_preserve_dtypes`.
- Make sure it does not nothing if the regressor does not define the `preserves_dtype` estimator tag.
- Identify a few regressors where this property holds, add the tag for `float64` and `float32` and check that the common test pass with:

```
pytest -vk "check_regressor_preserve_dtypes and RegressorClassName"
```

Once done, create a meta-issue to track all regressors that should be labeled to preserve dtypes, similarly to what the following issue does for transformers:

- #11000
user: ogrisel
labels: module:test-suite: everything related to our tests
float32: Issues related to support for 32bit data
issue_22644: state: open
title: Add `as_frame` to `sklearn.datasets.fetch_20newsgroups`
body: ### Describe the workflow you want to enable

I would like to load the 20 newsgroups dataset as a dataframe in one line of code:
```
X, y = sklearn.datasets.fetch_20newsgroups(return_X_y=True, as_frame=True)
```

### Describe your proposed solution

Add the argument `as_frame` to `sklearn.datasets.fetch_20newsgroups` to make the function behave similar to other data loading functions such as [`fetch_openml`](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_openml.html) or [`fetch_covtype`](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_covtype.html#sklearn.datasets.fetch_covtype).

### Describe alternatives you've considered, if relevant

Manually casting to a dataframe:
```
In [11]: X, y = sklearn.datasets.fetch_20newsgroups(return_X_y=True)                                                                                                                                         

In [12]: type(X)                                                                                                                                                                                             
Out[12]: list

In [15]: import pandas as pd                                                                                                                                                                                 

In [19]: pd.DataFrame(X, dtype='str')                                                                                                                                                                        
Out[19]: 
                                                       0
0      From: lerxst@wam.umd.edu (where's my thing)\nS...
1      From: guykuo@carson.u.washington.edu (Guy Kuo)...
2      From: twillis@ec.ecn.purdue.edu (Thomas E Will...
3      From: jgreen@amber (Joe Green)\nSubject: Re: W...
4      From: jcm@head-cfa.harvard.edu (Jonathan McDow...
...                                                  ...
11309  From: jim.zisfein@factory.com (Jim Zisfein) \n...
11310  From: ebodin@pearl.tufts.edu\nSubject: Screen ...
11311  From: westes@netcom.com (Will Estes)\nSubject:...
11312  From: steve@hcrlgw (Steven Collins)\nSubject: ...
11313  From: gunning@cco.caltech.edu (Kevin J. Gunnin...

[11314 rows x 1 columns]

```

### Additional context

The [string dtype](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.StringDtype.html#pandas.StringDtype) has been added in version 1.0 and the [minimal version for pandas in sklearn 1.1 will be 1.0.5](https://scikit-learn.org/dev/install.html#installing-the-latest-release) so I believe this should be possible to add.

CC @louquinze
user: mfeurer
issue_22626: state: open
title: Add an average callable parameter to sklearn.neighbors.KNeighborsRegressor
body: ### Describe the workflow you want to enable

For some algorithms of kNN it is desired to use other calculations at the end of this method besides using the mean. In some cases using a geometric average or arithmetic-geometric average is needed instead of the regular average.

### Describe your proposed solution

Add a parameter to the KNeighborsRegressor class, for computing the average at the end of the predict() method - something like:

```
# Use the default way to compute the average with a new parameter and string value of 'mean' which would be the default

KNeighborsRegressor(n_neighbors=5, metric=manhattan_distance, weights='uniform', average='mean')
```


```
# Allow callables for the new average parameter...

def geometric_mean(l):
    k=1
    for i in l:
        k*=i
        
    return pow(k, 1/len(l))

KNeighborsRegressor(n_neighbors=5, metric=manhattan_distance, weights='uniform', average=geometric_mean)





# Allow callables for the new average parameter...

def arithmetic_geometric_mean(l):
    tolerance=1e-10
    a0 = mean(l)
    g0 = geometric_mean(l)
    
    an, gn = (a0 + g0) / 2.0, math.sqrt(a0 * g0)
    while abs(an - gn) > tolerance:
        an, gn = (an + gn) / 2.0, math.sqrt(an * gn)
    
    return an

KNeighborsRegressor(n_neighbors=5, metric=manhattan_distance, weights='uniform', average=arithmetic_geometric_mean)
```

### Describe alternatives you've considered, if relevant

Not use scikit-learn's KNeighborsRegressor and build my own:

```
#
# Here avg_func is the callable method to compute the average
#
def kNN_regressor(q, R, truth, k, avg_func=my_mean, distance_func=euclidean_distance, **kwargs):
    idx = [0]*k
    dist = [0]*k
    
    for i in range(k):
        idx[i] = i
        if 'minkowski_param' in kwargs:
            mink_p = kwargs['minkowski_param']
            dist[i] = distance_func(q, R[i], mink_p)
        else:
            dist[i] = distance_func(q, R[i])
        
    max_idx = dist.index(max(dist))
        
    for i in range(k, len(R)):
        if 'minkowski_param' in kwargs:
            mink_p = kwargs['minkowski_param']
            d = distance_func(q, R[i], mink_p)
        else:
            d = distance_func(q, R[i])

        if d < dist[max_idx]:
            dist[max_idx] = d
            idx[max_idx] = i
            max_idx = dist.index(max(dist))
    
    t = []
    for i in range(k):
        t.append(truth[idx[i]])
        
    return avg_func(t)
```

### Additional context

_No response_
user: raymondj-pace
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_22625: state: open
title: Add zero_division None or np.nan
body: ### Describe the workflow you want to enable

This would be an extension of my previous addition https://github.com/scikit-learn/scikit-learn/pull/14900, where I added the parameter `zero_division` for precision, recall, f1. Afterwards, it was added for jaccard as well. 

Here, we would return np.nan when the metric is undefined. For the user to force this, she would pass `zero_division=None` or `zero_division=np.nan`. To be clear, **in both cases**, the result would be `np.nan` (even if passing `None`). 

This would be useful if the metric is used downstream for some average, to be excluded, as e.g. pd.mean does. In case the metric included an average, then such behaviour would already be implemented here, ignoring the figures being undefined.

### Describe your proposed solution

Allow zero_division to be `None` or `np.nan`. In both cases, disable the warning as we already do for `zero_divsion` 0 and 

Behaviour for each metric when passing either `None` or `np.nan` as follows. Keep same as today for other values, i.e. back-compatible

Precision:
* If pred_sum = 0, return `np.nan`
* If average != None, ignore from average any class metric being undefined

Recall:
* If true_sum = 0, return `np.nan`
* If average != None, ignore from average any class metric being undefined

F-score:
* if `beta=inf`, return recall, and `beta=0`, return precision
* elif precision=0 or recall=0 (or both), return 0. 
* else return `np.nan`
* If average != None, ignore from average any class metric being undefined

Jaccard:
* if all labels and pred are 0, return `np.nan`
* If average != None, ignore from average any class metric being undefined

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

https://github.com/scikit-learn/scikit-learn/issues/14876
https://github.com/scikit-learn/scikit-learn/pull/14900
user: marctorsoc
issue_22606: state: open
title: ENH Uses __sklearn_tags__ for tags instead of mro walking
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes https://github.com/scikit-learn/scikit-learn/issues/20804

#### What does this implement/fix? Explain your changes.
This PR implements `__sklearn_tags__`, while also keeping backward compatibility. The idea is to **not** walk the MRO anymore and use Python inheritance to get the tags. This means third party estimators needs to call `super().__sklearn_tags__`, create a new dictionary and return it.

#### Any other comments?
I suspect the current design was to allow third party developers to define `_more_tags` without the complete set of tags. `_safe_tags` will infer the missing tags with the default ones. If we want to support this use case, then `__sklearn_tags__` can also return a subset of the tags, and we have `_safe_tags` infer the missing ones with the defaults.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
labels: Needs Decision - API: Requires decision regarding API
issue_22574: state: open
title: ENH add Naive Bayes Metaestimator `ColumnwiseNB` (aka "GeneralNB")
body: **This PR is _ready_ for review.**
**Dear reviewer, please don't be discouraged by the amount of text below! The enumerated comments are optional and simply document certain considerations that I had while implementing this PR.**

Reference Issues/PRs
Fixes #12957 #15077 (issues)
Supersedes #16281 (stalled PR)
Somewhat related to #10856
Incorporates changes from PR #22565 , where I add abstract methods in `_BaseDiscreteNB`

#### What does this implement/fix? Explain your changes.
This implements a meta-estimator that mixes existing naive Bayes classifiers: `GaussianNB`, `BernoulliNB`, and others. The interface is inspired by `ColumnTransformer`, as was previously suggested by  @timbicker , @jnothman and others:
```python
>>> import numpy as np
>>> rng = np.random.RandomState(1)
>>> X = rng.randint(5, size=(6, 100))
>>> y = np.array([0, 0, 1, 1, 2, 2])
>>> from sklearn.naive_bayes import MultinomialNB, GaussianNB, ColumnwiseNB
>>> clf = ColumnwiseNB(nb_estimators=[('mnb1', MultinomialNB(), [0, 1]),
...                                   ('mnb2', MultinomialNB(), [3, 4]),
...                                   ('gnb1', GaussianNB(), [5])])
>>> clf.fit(X, y)
ColumnwiseNB(nb_estimators=[('mnb1', MultinomialNB(), [0, 1]),
                         ('mnb2', MultinomialNB(), [3, 4]),
                         ('gnb1', GaussianNB(), [5])])
>>> print(clf.predict(X))
[0 0 1 0 2 2]
```

The meta-estimator combines multiple naive Bayes estimators by expressing the overall joint probability ``P(x,y)`` through ``P(x_i,y)``, the joint probabilities of the subestimators: 
> ``Log P(x,y) = Log P(x_1,y) + ... + Log P(x_N,y) - (N - 1) Log P(y)``,  

where ``N`` is the number of estimators (3 in the example). The joint probabilities ``P(x_i,y)`` are exposed through the private method `_joint_log_likelihood(self, X)` in any instance of `_BaseNB` class, which is a parent to all naive Bayes classifiers implemented so far, including the proposed meta-estimator.

See further equations and detailed explanation [inside the collapsable section in this comment](https://github.com/scikit-learn/scikit-learn/pull/22574#issuecomment-1561703731).

#### Any other comments?

1. **About the name**. This meta-estimator was referred to as `GeneralNB` in the past, cf. stalled PR #16281. I think terms like "general" or "generalized" are too ambiguous. For example, Bernoulli and Gaussian distributions are both members of the general exponential family of distributions -- is this the generalisation we refer to? Another generalisation is having different distribution families for each class, e.g., `P(x|y=0)~N` and `P(x|y=1)~Mult`, or working with any user-provided distribution. In addition, I don't recall seeing "the general naive Bayes" in the literature: all models were _just naive Bayes_ with further assumptions of various degree of generality. 
In contrast, `ColumnwiseNB` is specific: it tells the user that something is going on with columns and forms a very appropriate association with `ColumnTranformer`. I think this is a much better name, and maybe there are even better options. I'll be fine with any decision, but I think `ColumnwiseNB` would be a good choice.
2. **Docstring** and **tests** are included.
3. **Use case**, **examples** and **guide**. I don't know any good use case or have any good example at hand. I'd be grateful for a suggestion ~~or if someone wrote an example using my implementation, while testing it at the same time. I guess, a paragraph or two should be added to the Guide as well. Shall we leave all this for another PR?~~
_[Updated on 2022-04-10]_ A section to naive Bayes guide is added. An auto-example (titanic dataset) is created for the gallery.
4. **Parallel jobs**. I implemented `fit` and `partial_fit` following `_BaseDiscreteNB` and took `ColumnTransformer`'s implementation as a blueprint for parallelism. I am not very familiar with the inner workings of `joblib`, and I ask reviewers to pay extra attention here. 
[Related [comment by jnothman](https://github.com/scikit-learn/scikit-learn/pull/22574#issuecomment-1080243263): _"... I suspect that fitting also will not benefit much from parallelism, but this may be more of an empirical question."_]
5. **MetaEstimator**. `_BaseComposition` is a parent class and get/set methods for parameters have been implemented and tested. Subestimators are cloned prior to fitting. 
6. **Callable columns** are supported. 
7.  **Pandas DataFrame** and string indexing of columns is supported.
8. There is no **``remainder``** parameter (un)like in ``ColumnTransformer``. It complicates the logic a little bit. Do you think it'd be really nice to have it? Maybe leave it till the next PR then.
9. **Repeated columns** are allowed and have the same effect as if they were duplicated.
10. `GaussianNB` and `_BaseDiscreteNB` use different conventions. For example, `priors` vs `class_prior` as hyperparameters and `class_log_prior_` vs `class_prior_` as attributes. I do not know which convention is more preferable. Just a remark.
11. The meta-estimator duplicates some functionality of its subestimators, such as checking feature names, sample weights, classes, supplied prior, counting and calculating priors as relative frequencies. I haven't found a way to avoid this.
12. (I forgot it)
13.  **Theoretical issue**. It is implicitly assumed in the equation ``Log P(x,y) = Log P(x_1,y) + ... + Log P(x_N,y) - (N - 1) Log P(y)``,  that the class log priors are finite and agree between the estimators and the subestimator: ``- inf < Log P(y) = Log P(y|1) = ... = Log P(y|N)``. The meta-estimators does not check this condition. Meaningless results, including ``NaN``, may be produced by the meta-estimator if the class priors differ or contain a zero probability. Usually this is not a problem, because all children of `_BaseNB` calculate class priors _directly_ as relative frequencies; unseen classes from `partial_fit` are on the user, not on us. But in general, class priors can be estimated differently or provided by the user in subestimators and the meta-estimator independently. We can distinguish 2 issues. First, what if the subestimators' class priors do not agree with the meta-estimator's prior? Then we could "marry" the subestimators not "at joint probabilities", but "at conditional probabilities", just as naive Bayes prescribes:  ``Log P(x|y) = Log P(x_1|y) + ... + Log P(x_N|y)`` or equivalently ``Log P(x,y) = Log P(x_1,y) - Log P(y|1) + ... + Log P(x_N,y) - Log P(y|N) + Log P(y)``, where  ``Log P(y|i)`` is the class prior from the ith subestimator. Second, what if some prior probabilities are zero? A problem arises only when the the zero found in the meta-estimator's class prior. This is something to think about. I wonder if someone came across a paper or a textbook that discusses and resolves this issue. [**Update**: see most recent comments [**1**](https://github.com/scikit-learn/scikit-learn/pull/22574#discussion_r1230299484) and [**2**](https://github.com/scikit-learn/scikit-learn/pull/22574#issuecomment-1592242979)]

Finally, I would appreciate any comments and suggestions, especially those addressing mistakes or aimed at efficient programming and "good code". I am glad to be learning from this community.

#### _**Updated on Wednesday, February 23, 2022 05:57:39 UTC**_:

14. **`estimatorNBs` vs `estimators`**. Initially, the parameter's name was "estimators", by analogy with "transformers". It turns out that many common tests failed simply because they duck-type models based on the presence of `estimators` in the parameter list. As a result, the tests assume that they're dealing with something like VotingClassifier and feed it LogisticRegression--something that ColumnwiseNB is not suppose to accept. ~I took the path of the least resistance and renamed `estimators` to `estimatorNBs`.~ [**Further update.** Following jnothman's [comment](https://github.com/scikit-learn/scikit-learn/pull/22574#issuecomment-1080030147), the parameter was later renamed to `nb_estimators`. glemaitre [thinks](https://github.com/scikit-learn/scikit-learn/pull/22574#discussion_r1058392780) the name should be `estimators`. [My comment](https://github.com/scikit-learn/scikit-learn/pull/22574#discussion_r1063946551) summarises my thoughts and links to the duck-typing issue.]

15. ~~The only change I made to pre-existing **test files** is adding `ColumnwiseNB` to `VALIDATE_ESTIMATOR_INIT`, a list that exempts `ColumnTransformer` and other estimators from a certain test.~~ _[Updated on 2022-05-29]_ The problematic test is passed by catching the error when `_estimators` setter in `ColumnwiseNB` fails to unpack parameters passed by the test. See #22537 for the similar change in `ColumnTransformer`.

#### _**Updated on Wednesday, February 23, 2022 09:14:13 UTC**_:

16. **pytest, joblib, and output**. I don't understand how pytest interacts with joblib, and why I can't capture any stdout produced within a parallel loop when running `pytest`. As a result, I cannot pytest such an output. Could someone please explain this unexpected behaviour or suggest a solution?

#### _**Updated on Sunday, February 27, 2022 09:00:13 UTC**_:

17. **`n_features_in_`**.  ~~Advice needed. First, I do not fully understand the purpose of this attribute and whether it is needed given we use `feature_names_in_`~~. ~~Second, setting it up using `BaseEstimator._check_n_features` can be problematic, since this method expects a "converted array", and I currently avoid converting `X` in the meta-estimator~~. 
_[Updated on 2022-10-04]_: Although `BaseEstimator._check_n_features` is typically passed _converted_ arrays (e.g., from `BaseEstimator._validate_data` or `ColumnTransformer.fit_transform`), it seems to work fine with pandas dataframes too. All examples work and all tests are passed.

#### _**Updated on Thursday, April 7, 2022 17:14:08 UTC**_:

18. There could be an advantage in taking `VotingClassifier` as a prototype instead of `ColumnTransformer`. The difference is that we would be passing a list of tuples `(str, estimator)` instead of `(str, estimator, columns)`. In the former case, `estimator` would have to be a base naive Bayes estimator wrapped together with a column selector, e.g. a `GaussianNB` + a selector of float columns. In fact, we could've redefined all `_BaseNB` to add this new parameter for column selection, whose default value selects all columns of `X`. The advantage is that now the subestimators' columns subsets are treated as separate hyperparameters for the purpose of `GridSearch`. We could write 
    ```param_grid = {"clf__gnb1__columns": [[5], [5, 6]]}```
    instead of much more verbose 
    ```
    param_grid = {"clf__estimators": [
    [('mnb1', MultinomialNB(), [0, 1]),
     ('mnb2', MultinomialNB(), [3, 4]),
     ('gnb1', GaussianNB(), [5])],
    [('mnb1', MultinomialNB(), [0, 1]),
     ('mnb2', MultinomialNB(), [3, 4]),
     ('gnb1', GaussianNB(), [5, 6])]}
    ]
    ```

#### _**Updated on Monday, April 11, 2022 05:30:23 UTC**_: 
19. **HTML representation** `ColumnwiseNB` displays subestimators in parallel, just like `ColumnTransformer`.


#### _**Updated on Thursday, July 7, 2022 23:24:45 UTC**_: 
20. **_validate_params** aka **common parameter validation**. Necessary changes towards #23462 are made.  A custom replacement for `test_common.py::test_check_param_validation` is implemented in `test_naive_bayes.py::test_cwnb_check_param_validation`. It is needed because `utils.estimator_checks._construct_instance` does not know how to create an instance of ColumnwiseNB, which leads to `test_common.py` skipping this test for ColumnwiseNB (without a notification by `pytest`; a message is displayed only when the test is called directly). ColumnTransformer, Pipeline, and a handful of other estimators suffer from this problem too.

#### _**Updated on Friday, October 7, 2022 16:26:00 UTC**_:
21. **Why **`ColumnTransformer`** is not enough?** This question was asked at a Discord voice meeting. First, `ColumnTransformer` is a transformer, not a classifier. Second, `ColumnTransformer` simply concatenates the transformed columns, whereas the naive Bayes requires additional calculations. Importantly, these additional calculations are done not on the subestimators' standard *output* (`predict`, `predict_proba`, `predict_log_proba`), but on the *intermediate* quantities (`predict_joint_log_proba`, formerly known as `_joint_log_likelihood`), see [this discussion](https://github.com/scikit-learn/scikit-learn/pull/22574#discussion_r889601021). This is why `ColumnTransformer` cannot be used even with additional logic wrapped on top of it.
user: avm19
issue_22573: state: open
title: ENH: Add refit callable generator for model sub-selection with Grid/RandomSearchCV
body: @jnothman @NicolasHug @amueller 

Hopefully you remember this one. Here are some further enhancements/fixes:
1) includes signed rank-sum hypothesis testing; 
2) the `se` rule is now implemented as standard-error rather than just SD as in the example `plot_grid_search_refit_callable.py`; 
3) more mature error-handling is implemented;
4) boilerplate and business logic has been removed, cognitive complexity reduced;
5) at reviewer's request, what was formerly a single class has now been organized into its own dedicated module;
6) docstring, doctesting, commenting, and comprehensive unit testing is now present (see `test_refit_callable_` in model_selection/tests/test_subselect.py
7) the `plot_grid_search_refit_callable.py` has also now been updated to leverage the newly added `_subselect` tools and explore the statistical advantages to using them w.r.t. mitigating train-test deviance and promoting greater tuning generalizability.
8) documentation for grid_search has been updated to add brief background on the theory and explanation of usage.

## Reference Issues/PRs
https://github.com/scikit-learn/scikit-learn/blob/master/examples/model_selection/plot_grid_search_refit_callable.py
https://github.com/scikit-learn/scikit-learn/blob/master/examples/model_selection/plot_grid_search_digits.py
https://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_digits.html
See also https://github.com/scikit-learn/scikit-learn/issues/11269. See also https://github.com/scikit-learn/scikit-learn/pull/11354. See also https://github.com/scikit-learn/scikit-learn/issues/12865. See also https://github.com/scikit-learn/scikit-learn/issues/9499. 

## What does this implement/fix? Explain your changes.
This PR formalizes methods for generating custom refit callables that can be used with `GridSearchCV` and `RandomSearchCV`. This new feature was partly motivated by discussions with other members of the Scikit-Learn team who expressed interest and assisted in developing the initial prototype. Since then, it has seen considerable renewed interest from other users in the scikit-learn and Kaggle communities, many of whom were surprised that the refit callable method for the "One Standard Error Rule" wasn't already implemented in the package. Rather than contribute that functionality specifically, which could arguably be too opinionated, I've attempted to generalize that strategy so that users can exploit `refit` for "model (sub)selection" based on any of a number of strategies.

## Any other comments?
- McNemar’s test, Shapiro-Wilk and others might be nice additions beyond Wilcoxon down the road but I think would go beyond the scope of this PR. Other custom selectors like these might be best left to the user to create, or if there is sufficient community interest, added formally to the core collection in `_subselect.py` through future PR's.

@dPys
user: dPys
issue_22561: state: open
title: Modifications to LinearRegression documentation.
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #22551.

#### What does this implement/fix? Explain your changes.

From the documentation of https://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares, one could understand that LinearRegression fits a model where the intercept $w_0$ is absent, that is, where $w_0=0$.

To prevent this misinterpretation, the present PR rephrases:

1. "Across the module, we designate the vector $w = (w_1, ..., w_p)$ as coef_ and $w_0$ as intercept_." as "Across the module, we designate the vector $(w_1, ..., w_p)$ as coef_ and $w_0$ as intercept_." 
2. "LinearRegression fits a linear model with coefficients $w = (w_1, ..., w_p)" as "LinearRegression fits a linear model with coefficients $w = (w_0, w_1, ..., w_p)" 
3. "and will store the coefficients of the linear model in its coef_ member" as "and will store the coefficients of the linear model in its coef_ and intercept_ members"

#### Any other comments?
user: thomasoliveira
labels: Needs Decision: Requires decision
issue_22556: state: open
title: ColumnTransformer: integer column index in dataframes unexpected behaviour and error (column selectors vs _get_column_indices)
body: ### Describe the bug

Unexpected behaviour when using integer column index in a DataFrame, other than natural ordering [0, 1, ...].

```
select_int = make_column_selector(dtype_include=np.int_)
ct = ColumnTransformer([('t2', Normalizer(norm="l1"), select_int)])
df1 = pd.DataFrame({'1': [1, 2, 3], '2': [9, 8, 7]})
df2 = pd.DataFrame({1: [1, 2, 3], 2: [9, 8, 7]})
ct.fit_transform(df1) # OK
ct.fit_transform(df2) # IndexError
```

The only difference between `df1` and `df2` is the type of column index. In my opinion, the results for these dataframes must be similar, but an error is raised for the latter.

As far as I could see, the problem stems from semantic ambiguity as to when to use `iloc`-based indexing vs `loc`-based indexing. In `_get_column_indices` [L382](https://github.com/scikit-learn/scikit-learn/blob/998e8f2068c7d9f43a30d997b63e4875688d0b1a/sklearn/utils/__init__.py#L382) this decision is based on the type of index and not on the type of the array. Whichever criterion is chosen, if it followed consistently in column selectors, the error shall be avoided. Probably.

### Steps/Code to Reproduce

(See above)

### Expected Results

(See above)

### Actual Results

(See above)

### Versions

```shell
Python dependencies:
          pip: 22.0.3
   setuptools: 60.8.1
      sklearn: 1.1.dev0
        numpy: 1.22.2
        scipy: 1.8.0
       Cython: 0.29.27
       pandas: 1.3.5
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 3.1.0

commit b28c5bba66529217ceedd497201a684e5d35b73c (upstream/main, origin/main, origin/HEAD, main)
Author: Thomas J. Fan 
Date:   Tue Feb 15 11:46:54 2022 -0500

    FIX DummyRegressor overriding constant (#22486)
```

user: avm19
labels: Needs Decision - API: Requires decision regarding API
issue_22551: state: open
title: LinearRegression in User Guide seems to have $w_0=0$
body: From the documentation of https://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares, one could understand that LinearRegression fits a model where the intercept $w_0$ is absent, that is, where $w_0=0$.

To prevent this misinterpretation, I suggest rephrasing (i) "LinearRegression fits a linear model with coefficients $w = (w_1, ..., w_p)" as "LinearRegression fits a linear model with coefficients $w = (w_0, w_1, ..., w_p)" AND (ii) "and will store the coefficients of the linear model in its coef_ member" as "and will store the coefficients of the linear model in its coef_ and intercept_ members"
user: thomasoliveira
issue_22538: state: open
title: Implementation of XRAY solver for NMF per Kumar 2013
body: ### Describe the workflow you want to enable

I was inspired by this talk [https://www.youtube.com/watch?v=kSfwY68gQ9I](url) to attempt an implementation of the XRAY solver for NMF per Kumar 2013: [https://arxiv.org/pdf/1210.1190.pdf](https://arxiv.org/pdf/1210.1190.pdf) and [https://arxiv.org/pdf/1312.7167.pdf](https://arxiv.org/pdf/1312.7167.pdf)

The talk and the paper make some pretty extraordinary claims about uniqueness, speed, and convergence - subject to certain assumptions about the matrix to be factorized.  

Here is the issue - the paper meets the criteria for inclusion, except for the number of citations, which is at 166 according to Google Scholar.  I would be happy to make a PR, but before I expend the effort I'd like a general sense from the maintainers whether the PR is likely to survive *assuming* the paper's claims pan out and the XRAY solver is a significant advancement for sklearn's NMF implementation.

I have reviewed open issues and cannot find any other proposals for implementing the XRAY solver.   I have, however, found a huge open PR for MinibatchNMF, issue #13308.  Presumably once this is released (milestoned for 1.1), the XRAY solver would be supported in that implementation as well.  I mention this only because I worry about that PR dogpiling on top of whatever work I would also do, causing headaches all around.

### Describe your proposed solution

I propose XRAY as an additional solver within the current NMF implementation, alongside cd and mu.  If there is a test for the separability or near-separability assumption, that may take the form of a warning or exception if the condition is not met.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: kyle-pena-nlp
issue_22523: state: open
title: Add rolling window to sklearn.model_selection.TimeSeriesSplit
body: ### Describe the workflow you want to enable

I wanted to ask whether any plans exist to implement a rolling/sliding window method in the TimeSeriesSplit class:

![68747470733a2f2f692e6962622e636f2f4b576b665137712f6e6577706c6f742e706e67](https://user-images.githubusercontent.com/28893120/154545460-d1629e8a-22a4-494d-affc-2df3cb95ade7.png)

 Currently, we are limited to using the expanding window type. For many financial time series models where a feature experiences a structural break, having a model whose weights are trained on the entire history can prove suboptimal.

I noted in https://github.com/scikit-learn/scikit-learn/pull/13204, specifically svenstehle's comments, that this might be on the horizon? 

### Describe your proposed solution

Current Implementation

```
>>> x = np.arange(15)
>>> cv = TimeSeriesSplit(n_splits=3, gap=2)
>>> for train_index, test_index in cv.split(x):
...      print("TRAIN:", train_index, "TEST:", test_index)
TRAIN: [0 1 2 3] TEST: [6 7 8]
TRAIN: [0 1 2 3 4 5 6] TEST: [ 9 10 11]
TRAIN: [0 1 2 3 4 5 6 7 8 9] TEST: [12 13 14]
```

Desired outcome

```
>>> x = np.arange(10)
>>> cv = TimeSeriesSplit(n_splits='walk_fw',  max_train_size=3, max_test_size=1)
>>> for train_index, test_index in cv.split(x):
...      print("TRAIN:", train_index, "TEST:", test_index)
TRAIN: [0 1 2] TEST: [3]
TRAIN: [1 2 3] TEST: [ 4]
TRAIN: [2 3 4] TEST: [5]
TRAIN: [ 3 4 5] TEST: [6]
TRAIN: [ 4 5 6] TEST: [7]
TRAIN: [ 5 6 7] TEST: [8]
TRAIN: [ 6 7 8] TEST: [9]
```

Where the 'stride' of the walk forward is proportionate to the test set, or walks by the max_train_size parameter?



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: cmp1
issue_22519: state: open
title: Tolerane parameter definiton in KMeans
body: ### Describe the issue linked to the documentation

KMeans documentation says:

> tol - Relative tolerance with regards to Frobenius norm of the difference in the cluster centers of two consecutive iterations to declare convergence.

But if we look into the code for stopping criterion (both in _kmeans_single_lloyd and _kmeans_single_elkan) we can see:
>                center_shift_tot = (center_shift ** 2).sum()
>                if center_shift_tot <= tol:

So it seems that stopping criterion is **nominal/direct** tolerance with regards to Frobenius norm instead of **relative**? Or maybe I am misinterpreting the code?


### Suggest a potential alternative/fix

_No response_
user: norbertryc
issue_22517: state: open
title: Add new feature to use only a sample of training data in KNNImputer
body: ### Describe the workflow you want to enable

I would like the team to add an argument(s) for using only part of the entire dataset to train the KNNImputer.

### Describe your proposed solution

This will require adding new argument(s) to the KNNImputer that allows the user to train the imputer with only part of the training data. This will save a significant amount of time as the current version takes too long and is almost not usable for fairly large training data.

The proposed solution runs orders of magnitude faster than using the whole dataset and may give higher accuracy than using constant imputation.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

imputer = KNNImputer()
imputer.fit(df)
df_new = imputer.transform(df) # This takes a very long time if dataset is fairly large

### What the proposed solution may look like
imputer = KNNImputer(sample=1000, random_state=None)
imputer.fit(df) # Uses only a thousand samples to train the imputer
df_new = imputer.transform(df) # Runs orders of magnitude faster than using the whole dataset.
user: abbey2017
issue_22504: state: open
title: Second build failure for s390x and ppc64
body: ### Describe the bug

Hi,
besides issue #22503 I've observed an _additional_ error in the test suite for the architectures s390x and ppc64.

### Steps/Code to Reproduce

```
pytest -m "not network" -v --color=no -k "not test_load_boston_alternative and not test_dump"
```
You can find a [full build log for arm64](https://buildd.debian.org/status/fetch.php?pkg=scikit-learn&arch=s390x&ver=1.0.2-1&stamp=1644952396&raw=0) where you can seek for the string above to see the start of the test procedure.

### Expected Results

Passing the test suite without any error.

### Actual Results

    ___________ [doctest] sklearn.feature_extraction._hash.FeatureHasher ___________
    081     DictVectorizer : Vectorizes string-valued features using a hash table.
    082     sklearn.preprocessing.OneHotEncoder : Handles nominal/categorical features.
    083
    084     Examples
    085     --------
    086     >>> from sklearn.feature_extraction import FeatureHasher
    087     >>> h = FeatureHasher(n_features=10)
    088     >>> D = [{'dog': 1, 'cat':2, 'elephant':4},{'dog': 2, 'run': 5}]
    089     >>> f = h.transform(D)
    090     >>> f.toarray()
    Expected:
        array([[ 0.,  0., -4., -1.,  0.,  0.,  0.,  0.,  0.,  2.],
               [ 0.,  0.,  0., -2., -5.,  0.,  0.,  0.,  0.,  0.]])
    Got:
        array([[ 0.,  0.,  0., -1.,  0.,  0.,  0.,  4.,  0.,  2.],
               [ 0.,  0.,  0., -2., -5.,  0.,  0.,  0.,  0.,  0.]])

    /<<PKGBUILDDIR>>/.pybuild/cpython3_3.9/build/sklearn/feature_extraction/_hash.py:90: DocTestFailure
    = 2 failed, 22017 passed, 404 skipped, 3 deselected, 59 xfailed, 38 xpassed, 2351 warnings in 686.07s (0:11:26) =


### Versions

```shell
`1.0.2`

Also see the [full build log](https://buildd.debian.org/status/fetch.php?pkg=scikit-learn&arch=s390x&ver=1.0.2-1&stamp=1644952396&raw=0) what other components are used.
```

user: tillea
labels: Needs Investigation: Issue requires investigation
issue_22485: state: open
title: [MRG] separate penalty factors for GLM regressions
body: #### Task list
- [x] Finished unit tests and added unit test coverage
- [x] Checked for PEP8 violations and ran black for code style
- [x] double check functionality using statsmodels

#### Reference Issues/PRs
https://github.com/scikit-learn/scikit-learn/issues/11566


#### What does this implement/fix? Explain your changes.
1. Added check for full rank design matrix when alpha = 0.0 (reverted due to comment from lorentzenchr)
2. Allows alpha to either be an iterable of non negative values in addition to being a scalar value
3. If alpha is an iterable then run checks on every value of alpha and also check length of alpha is equal to 2nd dimension of design matrix

Proof of correctness in script here:
https://github.com/MPCS-51042/final-project-xiaowei1234/blob/3b52d7ee67f2360aeace81b9479161b05c1c0ecb/proof.py

Link to output of proof script:
https://github.com/MPCS-51042/final-project-xiaowei1234/blob/3b52d7ee67f2360aeace81b9479161b05c1c0ecb/files/proof_output.log


user: xiaowei1234
issue_22453: state: open
title: Sensitivity Analysis function
body: 

## Proposal
Add a Sensitivity Analysis (SA) function.

The function would compute _Sobol'_ indices [1,2]. Consider a function `f` with parameters `x1`, `x2` and `x3`. Hence `y=f(x1,x2,x3)`. We are interested to know which parameter has the most impact, in terms of variance, on the value `y`.

I believe scikit-learn, and the wider scientific community, would greatly benefit to have such tool. I believe scikit-learn has something related with `feature_importances_` in some regressors.

As an expert in the field, I would propose to develop the initial functionalities and provide long term support.

## Background
_Sobol’_ indices are the cornerstone of SA and UQ and its application is not restricted to any field. It’s a non intrusive method which makes the only assumption that the variables are independent (this constraint can be alleviated).

Being able to compute sensitivity indices allows to reduce the dimensionality of a problem, better understand the importance of each factors and also see how parameters are interacting with each other. As such, it’s an important engineering tool. If you have 2 or more variables and only have the budget to improve your knowledge on one of them, this can help to make an informed choice. There are a lot of successful usage of SA in the literature and in real world applications. The EU (through the JRC), is now requiring to conduce uncertainty analysis when evaluating a system. They are recommending the use of _Sobol’_ indices.

## Example
Let’s take an example with the _Ishigami_ function:

`y = sin(x1) + 7*sin(x2)^2 + 0.1*x3^4*sin(x1)`

It’s not obvious to tell which variable would impact more `y`.

The Sobol’ indices are bounded from 0 to 1, with 1 meaning more important. Here they would be:

| Variable | First order Sobol’ | Total Sobol’ |
| -------- | ------------------ | ------------ |
|   x1     | 0.31               | 0.56         |
|   x2     | 0.41               | 0.44         |
|   x3     | 0.00               | 0.24         |


The difference between the first and total indices indicate an interaction between variables. The total indices allow to rank the variable by importance. `x1` is the most important. Looking at the first orders, `x3` by itself does not have an impact on the variance of the output. It’s its combination with another variable which makes it have a total impact of 0.24. `x1` in this case also have a difference in first and total indices while `x2` is the same. We can say that `x1` and `x2` have a second order interaction.

Computing the indices requires a large sample size, to alleviate this constraint, a common approach is to construct a surrogate model with Gaussian Process or Polynomial Chaos (to name the most used strategies). This allows to conduct a SA with a low computational budget (we see lots of engineering applications with expensive HPC codes taking advantage of this strategy).

## Implementation

Here is an example implementation I did a while back:

https://gist.github.com/tupui/09f065d6afc923d4c2f5d6d430e11696

Note that this implementation lack a few things such as higher order indices, other methods, input validation, doc, tests, other wrapping, etc. This is just to show how it works.

## Alternatives
Practitioners might be more familiarized with gradient based technics. Keywords include: gradient, adjoint. These methods are _local_ sensitivity analysis methods. And as such they are only capturing the sensitivity of the model in specific areas of the parameter space.

_Sobol’_ indices are variance based indices. There are other indices using higher moments, namely: _moment independant_ based sensitivity analysis. These methods are very attractive and provide lot of information while being simple to compute/analyse. They have been less studied but there is an increasing interest in the community. I believe this would make a good second step.

## References
Upon request, I can provide more information. Following are some fundational references–first 2 cited a few thousand times.

* [1] Sobol',I.M. (2001), _Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates_, MATH COMPUT SIMULAT,55(1–3),271-280, [doi:10.1016/S0378-4754(00)00270-6](http://dx.doi.org/10.1016/S0378-4754(00)00270-6)

* [2] Saltelli, A. et al., (2008), _Global Sensitivity Analysis. The Primer_, John Wiley & Sons, [doi:10.1002/9780470725184](http://dx.doi.org/10.1002/9780470725184)

* [3] Saltelli, A. et al., (2020), _The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support_, Environmental Modelling & Software, [doi:10.1016/j.envsoft.2020.104954](http://dx.doi.org/10.1016/j.envsoft.2020.104954)

For some examples of industrial applications (Part. 3), a visual explanation of some methods (Chap. 2.2) or more bibliography, you can also see my thesis:

https://github.com/tupui/PHD-Thesis/blob/master/phd_thesis_pamphile_roy.pdf

user: tupui
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_22446: state: open
title: `test_y_multioutput`in Gaussian Process is failing on Debian 32bit
body: ### Describe the bug

On 32bit systems on debian the test `test_y_multioutput` is failing.

The test will probably be just skipped during the build, this is not urgent, but maybe something underlying multioutput and Gaussian Process is hidden here (see #19995).

### Steps/Code to Reproduce

```
 pytest sklearn/gaussian_process/tests/test_gpr.py -k test_y_multioutput
```

### Expected Results

The test passes.

### Actual Results

```
=================================== FAILURES ===================================
______________________________ test_y_multioutput ______________________________
    def test_y_multioutput():
        # Test that GPR can deal with multi-dimensional target values
        y_2d = np.vstack((y, y * 2)).T
    
        # Test for fixed kernel that first dimension of 2d GP equals the output
        # of 1d GP and that second dimension is twice as large
        kernel = RBF(length_scale=1.0)
    
        gpr = GaussianProcessRegressor(kernel=kernel, optimizer=None, normalize_y=False)
        gpr.fit(X, y)
    
        gpr_2d = GaussianProcessRegressor(kernel=kernel, optimizer=None, normalize_y=False)
        gpr_2d.fit(X, y_2d)
    
        y_pred_1d, y_std_1d = gpr.predict(X2, return_std=True)
        y_pred_2d, y_std_2d = gpr_2d.predict(X2, return_std=True)
        _, y_cov_1d = gpr.predict(X2, return_cov=True)
        _, y_cov_2d = gpr_2d.predict(X2, return_cov=True)
    
        assert_almost_equal(y_pred_1d, y_pred_2d[:, 0])
        assert_almost_equal(y_pred_1d, y_pred_2d[:, 1] / 2)
    
        # Standard deviation and covariance do not depend on output
        assert_almost_equal(y_std_1d, y_std_2d)
        assert_almost_equal(y_cov_1d, y_cov_2d)
    
        y_sample_1d = gpr.sample_y(X2, n_samples=10)
        y_sample_2d = gpr_2d.sample_y(X2, n_samples=10)
        assert_almost_equal(y_sample_1d, y_sample_2d[:, 0])
    
        # Test hyperparameter optimization
        for kernel in kernels:
            gpr = GaussianProcessRegressor(kernel=kernel, normalize_y=True)
            gpr.fit(X, y)
    
            gpr_2d = GaussianProcessRegressor(kernel=kernel, normalize_y=True)
            gpr_2d.fit(X, np.vstack((y, y)).T)
    
>           assert_almost_equal(gpr.kernel_.theta, gpr_2d.kernel_.theta, 4)
E           AssertionError: 
E           Arrays are not almost equal to 4 decimals
E           
E           Mismatched elements: 2 / 3 (66.7%)
E           Max absolute difference: 4.77758733
E           Max relative difference: 27.70946804
E            x: array([ -4.6052,  -0.3981, -11.5129])
E            y: array([  0.1724,   0.4926, -11.5129])
sklearn/gaussian_process/tests/test_gpr.py:379: AssertionError
```

### Versions

```shell
scikit-learn 1.0.2

Python dependencies:
          pip: None
   setuptools: 59.6.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.7.3
       Cython: 0.29.24
       pandas: 1.3.5
   matplotlib: 3.5.1
       joblib: 0.17.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```
```

user: cmarmo
issue_22442: state: open
title: StandardScaler and PolynomialFeatures fail on zero-feature inputs during fit (should become passthrough)
body: ### Describe the bug

If you use StandardScaler or PolynomialFeatures (or other transformers, these are the two that hit me first) as elements in  a complex pipeline, an issue comes up if you ever fit these transformers on data that has no features (empty data). This happens a lot when you build a complex pipeline and you don't know what the features will be in advance. For example, in reinforcement learning problems, some problems will have context features while others won't, and we need pipelines for both the action and context features which are then concatenated (using FeatureUnion). My current workaround is that when I try to fit my pipeline, if it fails, it eliminates StandardScalers and PolynomialFeatures because they are a part of the context pipeline and we don't have context features for this problem, then fit again. 

### Steps/Code to Reproduce


from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.pipeline import Pipeline

pipeline = Pipeline([
    ('polynomial', PolynomialFeatures()),
    ('fully_transformed_action_and_context', StandardScaler()),
])

pipeline.fit([[]])

### Expected Results

No error is thrown. Pipeline becomes equivalent to passthrough.

### Actual Results

  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/pipeline.py", line 390, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/pipeline.py", line 348, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/joblib/memory.py", line 355, in __call__
    return self.func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/pipeline.py", line 893, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/base.py", line 847, in fit_transform
    return self.fit(X, **fit_params).transform(X)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/preprocessing/_polynomial.py", line 287, in fit
    _, n_features = self._validate_data(X, accept_sparse=True).shape
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/base.py", line 561, in _validate_data
    X = check_array(X, **check_params)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/utils/validation.py", line 806, in check_array
    raise ValueError(
ValueError: Found array with 0 feature(s) (shape=(1, 0)) while a minimum of 1 is required.


### Versions

```shell
System:
    python: 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18)  [Clang 6.0 (clang-600.0.57)]
executable: /usr/local/bin/python3.8
   machine: macOS-10.15.7-x86_64-i386-64bit
Python dependencies:
          pip: 20.0.2
   setuptools: 41.2.0
      sklearn: 1.0.1
        numpy: 1.19.5
        scipy: 1.4.1
       Cython: 0.29.16
       pandas: 1.4.0
   matplotlib: 3.2.1
       joblib: 0.14.1
threadpoolctl: 3.0.0
Built with OpenMP: True
```

user: douglasmason
issue_22438: state: open
title: Path for pluggable low-level computational routines
body: The goal of this issue is to discuss the design and prototype a way to register alternative implementations for core low level routines in scikit-learn, in particular to benefit from hardware optimized implementations (e.g. using GPUs efficiently).

## Motivation

scikit-learn aims to provide reasonably easy to maintain and portable implementations of standard machine learning algorithms. Those implementations are typically written in Python (with the help of NumPy and SciPy) or in Cython when the overhead of the Python interpreter prevents us to efficiently implement algorithms with (nested) tight loops. This allows us reasonably fast implementations as binary packages (installable with pip/PyPI, conda/conda-forge, conda/anaconda or Linux distros) for a variety of platforms (Linux / macOS / Windows) x (x86_64, i686, arm64, ppc64le) from a single code-base with no external runtime dependencies beyond Python, NumPy and SciPy.

Recently, GPU hardware have proven very competitive for many machine learning related workloads, either from a pure latency standpoint, or from the standpoint of a better computation/energy trade-off (irrespective of the raw speed considerations). However, hardware optimized implementation are typically not portable and mandates additional dependencies.

We therefore propose to design a way for our users to register alternative implementations of low-level computation routines in scikit-learn provided they has installed the required extension package(s) that matches their specific hardware.

## Relationship to adopting the Array API spec

This proposal is related and complementary to another effort, namely:

- #22352

The Array API spec is useful to make it possible to have some scikit-learn estimators written using a pure numpy syntax to delegate the computation to alternative Array API compatible libraries such as CuPy.

However, some algorithms in scikit-learn cannot be efficiently written using NumPy operation only, for instance the main [K-Means loop is written in Cython](https://github.com/scikit-learn/scikit-learn/blob/0de3b0d1eaacee9f7b15cabc05752cba945c7621/sklearn/cluster/_k_means_lloyd.pyx#L116-L152) to process chunks of samples in parallel (using `prange` and OpenMP), compute distance with centroids and reduce those distances to find assign each sample to its closest centroid on-the-fly while preventing unnecessary memory transfer between CPU cache and RAM.

If we want to run this algorithm efficiently on GPU hardware, one would need to dispatch the computation of this low level function to an alternative implementation that can work on GPU, either written in C/C++ with GPU-specific supporting runtime libraries and compilers (e.g. OpenCL, NVIDIA Cuda, Intel oneAPI DPC++, AMD ROCm...)  or using a Python syntax with the help of GPU support provided in numba for instance.

## List of candidate routines

- the main k-means loop (pairwise distances between samples and centroids with on-the-fly argmin reduction) 
- the core k-nearest neighbors computation loop (pairwise distances with on-the-fly arg-k-min reduction) 
- pairwise distances (without arg-k-min reduction)
- pairwise kernels computation (e.g. for use in Nystroem)
- ...

## Explicit registration API design ideas

I started to draft some API ideas in:

- https://hackmd.io/@4rHCRgfySZSdd5eMtfUJiA/S1RDJ3HCF

Feel free to comment here, or there.

This design is subject to evolve, in particular to make it possible to register both Array API extensions and non-Array API extensions with the same registration API.

## Next steps

- start a proof of concept implementation in a branch + external repo using either numba or C++ for k-means. Edit: this is happening here:
  - #25535
  - https://github.com/soda-inria/sklearn-numba-dpex
- once the API design has converged on the PoC, write a formal SLEP.
user: ogrisel
issue_22435: state: open
title: FEA post-fit calibration option in HGBT
body: ### Describe the workflow you want to enable

The histogram gradient boosted decision trees usually do not fulfil the so called *balance property* on the training data, i.e. `sum([proba]predictions) == sum(observations)`. A simple "post-fit" step could ensure this condition. This will usually decrease the in-sample performance, but I observed in practise that is is quite beneficial for the out-of-sample performance, in particular for non-canonical link-loss combinations such as the Gamma deviance with log link (with XGBoost or LightGBM as it is not yet available in scikit-learn). The main advantage is, however, better calibrated models, in-sample as well as out-of-sample.
```python
model = HistGradientBoostingRegressor()  # could also be a classifier
model.fit(X, y)
# post-fit recalibration to fulfil the balance property
model._baseline_prediction = link_function(np.mean(y) / np.mean(model.predit[_proba](X))
```
For quantiles, this would be slightly different.

### Describe your proposed solution

Add a new option `post_fit_calibration` (better name?!):
```python
model = HistGradientBoostingRegressor(post_fit_calibration=True)  # could also be a classifier
model.fit(X, y)
```

### Describe alternatives you've considered, if relevant

One could also invent a meta-estimator.

### Additional context

_No response_
user: lorentzenchr
issue_22434: state: open
title: PCA hangs occasionally when applied to data that fits in memory
body: ### Describe the bug

When trying to `fit` PCA with a large enough array that fits in memory (approximately `300 000 x 750`) with an aim to make about 3 times reduction, unstable results are obtained: in most cases the *fit succeeds*, but occasionaly it *hangs "forever"*.

Random seed is fixed, but unfortunalely it does not help in this case.
The `svd_solver` option is left at the default. Looks like explicitly specifying `svd_solver='arpack'` does not cause this issue.

### Steps/Code to Reproduce

```python
import numpy as np
import random

np.random.seed(42)
random.seed(42)

from sklearn.decomposition import PCA


if __name__ == '__main__':
    embeddings = np.random.RandomState(42).normal(size=(328039, 768))
    embeddings = embeddings.astype(np.float32)

    for i in range(20):
        print('Iteration', i)
        pca = PCA(n_components=260, random_state=42)
        pca = pca.fit(embeddings)

```

### Expected Results

Stable results: consistently successful fits over a finite time.

### Actual Results

Sometimes the process hangs, in most cases in the first 10 iterations. I haven't found any consistency pattern at what time it fails.

### Versions

```shell
System:
   python: 3.9.7 | packaged by conda-forge 
   machine: Windows-10-10.0.19043-SP0
   RAM: 32Gb

Python dependencies:
          pip: 21.3.1
   setuptools: 60.8.1
      sklearn: 1.0.1
        numpy: 1.20.1
        scipy: 1.7.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

user: IlyaKachan
labels: Needs Investigation: Issue requires investigation
issue_22433: state: open
title: DOC correct use of SAMPLE
body: ### Describe the issue linked to the documentation

The word "sample" in statistics means
> a set of individuals or objects collected or selected

see https://en.wikipedia.org/wiki/Sample_(statistics). Thus, a "single sample" (usually) already consists of several data points or observations (rows), i.e. a sample is a dataset.

In the scikit-lean documentation, however, "samples" is often used to denote the single data points (rows) in a given dataset. A few examples:
- From the glossary: "sample/samples: We usually use this term as a noun to indicate a single feature vector. Elsewhere a sample is called an instance, data point, or observation"
- `n_samples` means "sample size n" and not "there are n samples, i.e. n datasets".
- "However, by partitioning the available data into three sets, we drastically reduce the number of samples which can be used for learning the model ..." from https://scikit-learn.org/stable/modules/cross_validation.html
- "setting the minimum number of samples required at a leaf node" from https://scikit-learn.org/stable/modules/tree.html

### Suggest a potential alternative/fix

My personal point of view is that this is harmful, in particular for education. Therefore, ideally, this would be fixed, at least in the text. I'm well aware for the size of this change and I don't expect that API critical names are changed such as `n_samples` (even if that is unfortunate).
user: lorentzenchr
labels: Needs Decision: Requires decision
issue_22432: state: open
title: Some ideas for a developer-level API
body: Scikit-Learn developers have long been debating about improving the developer utilities (see #15801 and #20657). The current discussion revolves around the adoption of a **specific syntax** to differentiate the develop API from the common user API. Here, I'd like to propose an alternative solution: **package layering**.

The idea is to break scikit-learn into 2 separate packages, `sklearn` and `sklearn-core`, with the former depending on the latter and the latter providing common utilities for the former. The package `sklearn` defines the user-level API for common users, and the package `sklearn-core` defines the develop-level API for developers (including the third-party ones).

![drawing](https://user-images.githubusercontent.com/6860682/153353262-046b8840-f382-4a46-8356-3d9f7ab0f65f.svg)

I see this solution as a neater one than using a specific syntax. The decomposition of the package explicitly and unambiguously defines the develop API rather than implicitly. It also encourages the core developers to think more carefully when developing scikit-learn. In addition, it has an extra benefit for third-party packages: they can now depend on the light-weight `sklearn-core` than the heavy-duty `sklearn`.

user: WenjieZ
labels: Needs Decision: Requires decision
issue_22431: state: open
title: Correct precision at k
body: #### Reference Issues/PRs

Fixes https://github.com/scikit-learn/scikit-learn/issues/7343. 
Replaces PR-s https://github.com/scikit-learn/scikit-learn/pull/4975 and https://github.com/scikit-learn/scikit-learn/pull/14859.

#### What does this implement/fix? Explain your changes.

Implement precision@k metric for ranking problems.

#### Any other comments?

I've been doing recommendations and search for a few years now and I've been really disappointed that `sklearn` doesn't provide many mainstream ranking metrics, including precision@k.

The existing PRs (https://github.com/scikit-learn/scikit-learn/pull/4975 and https://github.com/scikit-learn/scikit-learn/pull/14859) look very sad, having no activity for years. I guess it's my turn to be eventually forgotten in the history of PR-s. Instead of taking over the unsuccessful attempts, I would like to push an alternative solution that focuses on correctness rather than performance.

The key features of this implementation differentiating it from the above-mentioned approaches:
1.  Correct treatment of ties in the provided scores.
     - The previous implementations aren't treating equal scores as equal. The worst outcome of this negligence is the score that a const classifier/ranker gets: imagine you want to calculate `precision_at_2(y_true=[0,0,1,1], y_score=[1,1,1,1])`. If you treat equal scores equally you should end up with the metric value of 0.5 which means that on average there is 1 relevant document out of 2. Otherwise you end up with misleading semi-random estimates, e.g. 1 - that will tell you that your constant classifer is perfect. For reference, this scenario is handled correctly in the `ndcg_score`, and I don't see why it shouldn't be here.
2. No support for sparse matrices
    - TBH I don't view it to be a requirement for the metric. As briefly mentioned in https://github.com/scikit-learn/scikit-learn/pull/4975 `y_score` is dense in most practical applications (at least in my experience). In my estimation this makes sparse matrix support of very little value. On top of that, the current implementation of such ranking metrics as `average_precision_score` and `ndcg_score` doesn't support sparse matrices either. This is another reason why I don't believe it's necessary.

When it comes to `recall@k`, although it's not as popular in ranking (at least in my biased experience), it can definitely be added re-using most of the code if we find this PR sensible.
user: ilivans
labels: Needs Decision: Requires decision
issue_22427: state: open
title: Improve the metrics API
body: ### Describe the workflow you want to enable

Currently metrics in scikit-learn have several shortcomings:
- Although it is possible to pass custom metrics as callables, they are called for each pair of inputs. This makes impossible to create custom metrics with the same performance as the existing ones (#22219).
- Every metric has different additional parameters. This requires that methods using them expose the particular parameters or provide a catch all `metric_params` parameter (and not all of them do that).
- Currently, metrics for types other than numeric vectors are not well supported (#15932).
- Some metrics, such as Mahalanobis, depend on the data. There is currently no way to use them in a cross-validation setting without incurring in data leakage (#6915).

### Describe your proposed solution

IMHO, metrics should be:
- Instances of classes, not functions. The additional parameters of the metrics (e.g. the `p` for Minkowski) could then be passed to the metric at creation time. Advantages: no need to homogeneize args, no need for `metric_params`. If the metrics inherit from `BaseEstimator`, the parameters could also be optimized in cross validation.
- They should accept two matrices with the same first dimension and compute all the paired distances. Advantage: no distinction between provided metrics and custom metrics: all of them could be implemented efficiently without multiple calls for each pair of vectors.
- There should be a way to implement an optimized version of pairwise distances. Although a default ok-ish version can be implemented by calling the function several times with the necessary matrices, for some metrics it is possible to have a better implementation. Either there should be a `Metric` ABC class with a default `pairwise` method that can be overriden, or a `singledispatch` function could be made for pairwise distances, so that one can register the implementation for their class. Advantages: again, allowing custom metrics to be implemented efficiently.
- Metrics know their input types. One should be able to provide metrics to compare strings or other custom types, and algorithms that only need to apply a metric, such as k-NN classification, should delegate the validation to the metric.
- (Optional) Some metrics, such as Mahalanobis, depend on the data. These metrics could have a `fit` method, allowing them to be used in cross-validation without incurring in data leakage.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: vnmabus
labels: Needs Decision - API: Requires decision regarding API
issue_22420: state: open
title: Y_trans is not equal to y_scores_ in PLSRegression
body: ### Describe the bug

The documentation that `Y_trans` obtained from `fit_transform` should be equal to `y_scores_` fitted attribute but this is not the case.

We should investigate if this is normal or not

### Steps/Code to Reproduce

```python
In [1]: from sklearn.cross_decomposition import PLSRegression
   ...: X = [[0., 0., 1.], [1.,0.,0.], [2.,2.,2.], [2.,5.,4.]]
   ...: Y = [[0.1, -0.2], [0.9, 1.1], [6.2, 5.9], [11.9, 12.3]]
   ...: pls2 = PLSRegression(n_components=2)
   ...: X_trans, Y_trans = pls2.fit_transform(X, Y)
```

### Expected Results

`Y_trans` should be equal to `y_scores_`.

### Actual Results

```python

In [2]: Y_trans
Out[2]: 
array([[-1.10089007,  2.64706486],
       [-1.73668766, -5.50753532],
       [ 1.05463999,  5.91255735],
       [ 1.78293774, -3.05208689]])

In [3]: pls2.y_scores_
Out[3]: 
array([[-1.40620988,  0.09323305],
       [-1.10143324, -0.85118366],
       [ 0.37266921,  1.64933181],
       [ 2.13497391, -0.8913812 ]])
```

### Versions

```shell
In [8]: sklearn.show_versions()

System:
    python: 3.8.12 | packaged by conda-forge | (default, Sep 16 2021, 01:38:21)  [Clang 11.1.0 ]
executable: /Users/glemaitre/mambaforge/envs/dev/bin/python
   machine: macOS-12.1-arm64-arm-64bit

Python dependencies:
          pip: 21.3
   setuptools: 58.2.0
      sklearn: 1.1.dev0
        numpy: 1.21.2
        scipy: 1.8.0.dev0+1902.b795164
       Cython: 0.29.24
       pandas: 1.3.3
   matplotlib: 3.4.3
       joblib: 1.2.0.dev0
threadpoolctl: 3.0.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /Users/glemaitre/mambaforge/envs/dev/lib/libopenblas_vortexp-r0.3.18.dylib
        version: 0.3.18
threading_layer: openmp
   architecture: VORTEX
    num_threads: 8

       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /Users/glemaitre/mambaforge/envs/dev/lib/libomp.dylib
        version: None
    num_threads: 8
```
```

user: glemaitre
issue_22417: state: open
title: Kernel Ridge Classifier
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.

Kernel Ridge Regression (the only Kernel Ridge estimator that is right now implemented in `master`) has a real simple mathematical formulation to adjust a regression problem using squared error loss combined with L2 regularization. It is similar in SVM is done, but with equal constrictions instead of inequal ones as in SVM.

Using a vector encoding, this regressor can be adapted as a multilabel classifier while keeping all the advantages over the kernel trick. I followed the mathematical explanation from section 3 in [Venkatesh's paper from 2007](http://dro.deakin.edu.au/eserv/DU:30044591/venkatesh-facerecognition-2007.pdf). Basically, it transforms the target with r labels into a vector with r components.

#### Any other comments?


This was proposed previously on this PR. Eventually, @ogrisel suggested opening it again https://github.com/scikit-learn/scikit-learn/pull/10633#issuecomment-820274818

Reviewing the code, I found out that this technique was already applied for the linear case, `sklearn.linear_model.RidgeClassifier`. So basically, this model uses the kernel trick, thus it can be applied to nonlinear ML problems.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: cperales
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_22411: state: open
title: Pooled variance for LDA estimators
body: ### Describe the workflow you want to enable

We use LDA a lot in ERP-BCI research. So far we use our custom implementation of LDA as it makes use of pooled variance, i.e., subtract class-wise means from `X` and use all data to calculate the within class covariance matrix. In practice, this often leads to better performance, especially when classes are unbalanced and only few data is available.

### Describe your proposed solution

If there is interest, I could push a PR that enables this behaviour for the `eigen` and `lsqr` solvers by using a new parameter, e.g., `use_pooled_cov=False` which is passed to `_class_cov` and if it is true, first subtract class means and then estimate the covariance on all data.
I am not aware if there is an equivalent implementation for the `svd` solver.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: jsosulski
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_22408: state: open
title: Deprecate the parameter `**kwds` in function `def paired_distances` since it is unused
body: ### References
#22380 

Deprecate the parameter `**kwds` in function `def paired_distances` since it is unused:  
`def paired_distances(X, Y, *, metric="euclidean", **kwds):`

_Originally posted by @reshamas in https://github.com/scikit-learn/scikit-learn/issues/22380#issuecomment-1031844726_
user: reshamas
labels: Needs Decision - API: Requires decision regarding API
issue_22402: state: open
title: Default hyperparameter values for novelty detection with LocalOutlierFactor and OneClassSVM
body: [Not sure whether this should be an issue, feel free to change it into a 'discussion'.]

We recently published a paper where we established optimal default hyperparameter values for a number of novelty detection algorithms, on the basis of 246 novelty detection problems derived from 50 UCI datasets.

https://arxiv.org/abs/2101.11037

For LocalOutlierFactor, we got the highest mean AUROC by using Manhattan distance and `k=2.5*np.log(n)`, where $n$ is the number of instances in the training data.

For OneClassSVM (using the RBF kernel), the optimal values were `nu=0.2` and `gamma=1/(0.25*m)`, where m is the number of features. Note that from what I understand, `gamma` is sensitive to scale; we scaled by the interquartile range (`RobustScaler`).

I mention this here so you can decide whether you want to change the default values used by scikit-learn, or mention it in your documentation. I realise that these classes are also used for unsupervised outlier detection, which may have different optimal default hyperparameter values, although the values we established may be a good proxy.

(We also introduced a new algorithm for novelty detection, Average Localised Proximity, which in particular outperforms LOF, but the paper doesn't (yet) satisfy your criteria for inclusion. For those interested, I've implemented it in my own library [fuzzy-rough-learn](https://github.com/oulenz/fuzzy-rough-learn).)
user: oulenz
labels: Needs Decision: Requires decision
issue_22383: state: open
title: Add friedman's H statistic
body: ### Describe the workflow you want to enable

It's a relatively straight-forward measure of whether there's significant interaction between variables in a model given the marginal and pairwise partial dependence.
We don't do a lot of statistics, but arguably this is more an interpretability feature.
See https://www.firmai.org/bit/interaction.html#theory-friedmans-h-statistic

### Describe your proposed solution

We could implement it either as a function or at least as an example to copy &paste.

### Describe alternatives you've considered, if relevant

People can always implement it themselves, but it might not be super straight-forward? We could also have an example, though.

### Additional context

_No response_
user: amueller
labels: Hard: Hard level of difficulty
issue_22366: state: open
title: Add attribute with target class pairs of each estimator in OneVsOneClassifier
body: ### Describe the workflow you want to enable

I'd like to suggest to add attribute to `OneVsOneClassifier` that will store pairs of classes used to train each estimator in `estimators` attribute. In this case it will be easier to map each estimator to it's target.

### Describe your proposed solution

As a solution, I propose to add attribute `target_pairs` or any other more suitable name that will store list of tuples of pairs. Also, it should be clear what class out of two is used as `1`and which one is used as `0` while training.

### Describe alternatives you've considered, if relevant

I couldn't find any information about how pairs are generated in documentation, so I opened a source code and found these lines in `fit` method which, I suppose, address this issue:

`estimators_indices = list(
    zip(
        *(
            Parallel(n_jobs=self.n_jobs)(
                delayed(_fit_ovo_binary)(
                    self.estimator, X, y, self.classes_[i], self.classes_[j]
                )
                for i in range(n_classes)
                for j in range(i + 1, n_classes)
            )
        )
    )
)`

### Additional context

_No response_
user: dimadgo11
issue_22363: state: open
title: Improving the runtime efficiency of the SVC for a precomputed kernel during inference
body: ### Describe the workflow you want to enable

When one wants to use a custom kernel in the `svm.SVC` class ([docs](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html)) there are two options: either to define a custom function and pass it as a callable to `svm.SVC` or to pre-compute the kernel matrix and use the `kernel=precomputed` argument of `svm.SVC`. From my understanding, following the class inheritance of `svm.SVC`: [source](https://github.com/scikit-learn/scikit-learn/blob/8cf09596bda0279d3e66d268ac9d0af878067718/sklearn/svm/_base.py#L224), these two options are computationally equivalent. _Please correct me if my assumption is wrong, because all the following points stem from that_.

In inference time, i.e. when you use `SVC.predict(X_test)`, with `kernel=precomputed`, the expected input matrix is of shape (n_test, n_train), where n_test and n_train are the number of test and train data samples, respectively. I believe there is an inefficiency because the above pre-computed matrix X_test includes more dot products than the ones needed to classify the test data points, since, the SVM model uses only the support vectors (subset of the training data vectors) to predict a label. To elaborate, a trained (fitted) SVM model <img src="https://render.githubusercontent.com/render/math?math=f"> classifies a test data point <img src="https://render.githubusercontent.com/render/math?math=\hat{x}"> in the following way (omitting the bias term),

<img src="https://render.githubusercontent.com/render/math?math=\hat{y}=\text{sign}[f(\hat{x})],\,\text{where}\, f(\hat{x})= \sum_{i=0}^{N_\text{sv}}\alpha_iy_iK(x_i,\hat{x})">, where <img src="https://render.githubusercontent.com/render/math?math=\alpha_i"> are the dual coefficients, <img src="https://render.githubusercontent.com/render/math?math=y_i "> are  the labels of the support vectors, <img src="https://render.githubusercontent.com/render/math?math=N_{SV}"> is the number of the support vectors identified during training, and <img src="https://render.githubusercontent.com/render/math?math=K"> is our kernel. The equation can be rewritten in a matrix form to accommodate classification of a set of test data points X_test, as mentioned above. 

All the dual coefficients that correspond to non-support data vectors are zero hence, the above sum is only over the support vectors. However, X_test has some columns that, in general, don't correspond to support vectors.

Isn't it more efficient to compute X_test with a shape of (n_test, `svm.SVC.n_support_`)? The only references to this issue that I was able to find are quite old (2012), by @amueller and @mblondel in #790 and https://github.com/scikit-learn/scikit-learn/pull/789#issuecomment-5259099. Nevertheless, no possible solution was discussed or implemented.

Can such an improvement be implemented? If not, I would really appreciate any information regarding the design choices that restrict such an implementation :)

### Describe your proposed solution

I wasn't able to pinpoint which parts of the source code could be changed to accommodate the above proposal. Instead, I will provide here a simple example of how the `svm.SVC` class could make use of the above enhancement if it is implemented.
```python
from sklearn.datasets import load_breast_cancer
from sklearn.svm import SVC
from sklearn.utils import shuffle
import numpy as np

dataset = load_breast_cancer()
X, y = shuffle(dataset.data,dataset.target)
X_train, X_test = X[:100, :], X[100:, :]
y_train, y_test = y[:100], y[100:]

svc = SVC(kernel='precomputed')

kernel_train = np.dot(X_train, X_train.T)  # simple example of a linear pre-computed kernel
svc.fit(kernel_train, y_train)
# What the current implementation requires:
kernel_test = np.dot(X_test, X_train.T)
# What would be more efficient:
kernel_test_ef = np.dot(X_test, X_train[svc.support_, :].T)

y_pred = svc.predict(kernel_test,y_test)# Raises an error if kerne_test_ef is used.		
													
```
An example of the shape difference between `kernel_test_ef` and `kernel_test`:
```python
svc.n_support_: [7 5]
kernel_test_eff.shape: (469, 12)
kernel_test.shape: (469, 100)
```
This difference can cause a significant runtime difference when scaling to large datasets.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: vbelis
labels: Hard: Hard level of difficulty
issue_22360: state: open
title: Support per target sample_weight in MultiOutputRegressor and MultiOutputClassifier
body: For multi-label label training if one of the targets has a NaN value, the row (with all the other targets) has to be discarded which is a bit problematic if there is a fair number of NaN in the targets.

A way around this could be to support  sample_weight to be not only a vector but also an array (n_samples, n_targets), same as y in the case of multi-label training. Then sample_weight could be used as a binary indicator whether the target row should be kept or not.

When training each model, one would take the appropriate column in the sample weight matrix.  It's a fairly simple change in 
https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09b/sklearn/multioutput.py#L140
but it might be a bit sensitive as technically it adds features  to the API of fit for those models.  Overall the fact that sample_weight can be the same shape as y seems reasonable to me, but we might not want to implement it in all models. MultiOutputRegressor  and MultiOutputClassifier being focused on multi-label training could a good compromise IMO.

WDYT? 

cc @lorentzenchr
user: rth
issue_22352: state: open
title: Path for Adopting the Array API spec
body: I have been experimenting with adopting the [Array API](https://data-apis.org/array-api/latest/index.html) spec into scikit-learn. The Array API is one way for scikit-learn to run on other hardware such as GPUs. 

I have some POCs on my fork for [LinearDiscriminantAnalysis](https://github.com/thomasjpfan/scikit-learn/pull/102) and [GaussianMixture](https://github.com/thomasjpfan/scikit-learn/pull/99). Overall, there is runtime performance benefit when running on CuPy compared to NumPy, as shown in notesbooks for [LDA (14x improvement)](https://gist.github.com/thomasjpfan/45f8bd908f56f6ab5107c0bdde04b7e7) and [GMM (7x improvement)](https://gist.github.com/thomasjpfan/effd04ef0909f0c59dd8c95fa64bbfa3). 

Official acceptance of the Array API in numpy is tracked as [NEP 47](https://numpy.org/neps/nep-0047-array-api-standard.html).

## Proposed User API

Here is the proposed API for dispatching. We require the array to adopt the [Array API standard](https://data-apis.org/array-api/latest/purpose_and_scope.html) and we have a configuration option to turn on Array API dispatching:

```python
# Create Array API arrays following spec
import cupy.array_api as xp
X_cu = xp.asarray(X_np)
y_cu = xp.asarray(y_np)

# Configure scikit-learn to dispatch
from sklearn import set_config
set_config(array_api_dispatch=True)

# Dispatches using `array_api`
lda_cu = LinearDiscriminantAnalysis()
lda_cu.fit(X_cu, y_cu)
```

This way the user can decide between the old behavior of potentially casting to NumPy and the new behavior of using the array api if available.

## Developer Experience

The Array API spec and the NumPy API overlaps in many cases, but there is API we use in NumPy and not in Array API. There are a few ways to bridge this gap while trying to keep a maintainable code base:

1. Wrap the Array-API namespace object to make it look "more like NumPy"
2. Wrap the NumPy module to make it look "more like ArrayAPI"
3. Helper functions everyone

**1 and 2 are not mutually exclusive**. To demonstrate these options, I'll do a case study on `unique`. The Array API spec does not define a `unique` function, but a `unique_values` instead.

### Wrap the Array-API namespace object to make it look "more like NumPy"

```python
def check_y(y):
    np, _ = get_namespace(y)  # Returns _ArrayAPIWrapper or NumPy
    classes = np.unique(y)

class _ArrayAPIWrapper:
    def unique(self, x):
        return self._array_namespace.unique_values(x)
```

Existing scikit-learn code does not need to change as much because the Array API "looks like NumPy"

### Make NumPy object "look more like Array-API"

```python
def check_y(y):
    xp, _ = get_namespace(y)  # Returns Array API namespace or _NumPyApiWrapper
    classes = xp.unique_values(y)

class _NumPyApiWrapper:
    def unique_values(self, x):
        return np.unique(x)
```

We need to update scikit-learn to use these new functions from the Array API spec.

### Helper functions everyone

```python
def check_y(y):
    classes = _unique_values(y)

def _unique_values(x):
    xp, is_array_api = get_namespace(x)
    if is_array_api:
        return xp.unique_values(x)
    return np.unique(x)
```

We need to update scikit-learn to use these helper functions when API diverges. Some notable functions that needs some wrapper or helper functions include `concat`, `astype`, `asarray`, `unique`, `errstate`, `may_share_memory`, etc.

For my POCs, I went with a mostly option 1 where I wrapped Array API to look like NumPy. (I did wrap NumPy once to get `np.dtype`, which is the same as `array.astype`).

CC @scikit-learn/core-devs 

<details>
 <summary><b>Other API considerations</b></summary>

- Type promotion is more strict with Array API

```python
import numpy.array_api as xp
X = xp.asarray([1])
y = xp.asarray([1.0])

# fails
X + y
```

- No method chaining. (Array API arrays do not have methods on them)

```python
(X.mean(axis=1) > 1.0).any()

# becomes
xp.any(xp.mean(X, axis=1) > 1.0)
```

- Array API has no concept of order
- Array API does not have integer indexing with `__getitem__`, alternative is `take` which is going into the Array API spec.
- [No views into arrays in the Array API spec](https://data-apis.org/array-api/latest/design_topics/copies_views_and_mutation.html)
- Can not support `Dask` or `JAX` at first because of they do not support methods that have [Data-dependent output shapes](https://data-apis.org/array-api/latest/design_topics/data_dependent_output_shapes.html) such as `unique`.

</details>
user: thomasjpfan
labels: Needs Decision - API: Requires decision regarding API
issue_22349: state: open
title: "Determinant has increased; this should not happen" in EllipticEnvelope
body: ### Describe the bug

Fitting the dataset given in the example below prints the following RuntimeWarning:

RuntimeWarning: Determinant has increased; this should not happen: log(det) > log(previous_det) (-6.080855240649472 > -46.216908559817156). You may want to try with a higher value of support_fraction (current value: 0.520).

It is not clear whether this is effectively an assertion error ("this should not happen"), i.e. a bug in the implementation; or a problem with the dataset (perhaps it's too degenerate for the fitter); that's OK too but the message's wording is a bit ambiguous.

### Steps/Code to Reproduce

```python
from sklearn.covariance import EllipticEnvelope

EllipticEnvelope().fit_predict([
    [ 13.6144901 , -7.67435074],
    [ 13.6144901 , -7.67435074],
    [-19.25379655, 10.8531709 ],
    [ 14.42724592, -5.86127761],
    [ 13.6144901 , -7.67435074],
    [ 14.42724592, -5.86127761],
    [ 13.6144901 , -7.67435074],
    [ 13.6144901 , -7.67435074],
    [ 13.6144901 , -7.67435074],
    [ 13.6144901 , -7.67435074],
    [ 14.42724592, -5.86127761],
    [-19.25379655, 10.8531709 ],
    [ 14.42724592, -5.86127761],
    [ 14.42724592, -5.86127761],
    [ 14.42724592, -5.86127761],
    [ 13.6144901 , -7.67435074],
    [ 13.54233675, -7.07064384],
    [ 13.05506301, -7.65279881],
    [ 14.42724592, -5.86127761],
    [ 14.42724592, -5.86127761],
    [ 13.54233675, -7.07064384],
    [ 13.05506301, -7.65279881],
    [ 13.05506301, -7.65279881],
    [ 13.05506301, -7.65279881],
    [ 13.54233675, -7.07064384],
    [ 13.05506301, -7.65279881],
    [ 13.05506301, -7.65279881],
    [ 13.05506301, -7.65279881],
    [ 13.54233675, -7.07064384],
    [ 14.42724592, -5.86127761],
    [ 13.05506301, -7.65279881],
    [ 14.42724592, -5.86127761],
    [ 14.42724592, -5.86127761],
    [ 13.6144901 , -7.67435074],
    [ 14.42724592, -5.86127761],
    [ 14.42724592, -5.86127761],
    [ 14.42724592, -5.86127761],
    [ 13.6144901 , -7.67435074],
    [ 13.05506301, -7.65279881],
    [ 13.6144901 , -7.67435074],
    [ 13.05506301, -7.65279881],
    [ 13.05506301, -7.65279881],
    [ 13.05506301, -7.65279881],
    [ 14.42724592, -5.86127761],
    [ 13.05506301, -7.65279881],
    [ 13.05506301, -7.65279881],
    [ 14.17049137, -7.69397   ],
    [ 14.17049137, -7.69397   ],
    [-19.25379655, 10.8531709 ],
    [ 13.6144901 , -7.67435074],
    [ 13.05506301, -7.65279881],
    [ 13.05506301, -7.65279881],
    [ 13.6144901 , -7.67435074],
    [ 13.6144901 , -7.67435074],
    [ 12.59641636, -9.37178184],
    [ 13.6144901 , -7.67435074],
    [ 13.05506301, -7.65279881],
    [ 13.6144901 , -7.67435074],
    [ 13.6144901 , -7.67435074],
    [ 14.42724592, -5.86127761],
    [ 13.05506301, -7.65279881],
    [ 13.6144901 , -7.67435074],
    [ 13.6144901 , -7.67435074],
    [ 14.42724592, -5.86127761],
    [ 14.17049137, -7.69397   ],
    [ 14.42724592, -5.86127761],
    [ 14.17049137, -7.69397   ],
    [ 14.42724592, -5.86127761],
    [ 14.42724592, -5.86127761],
    [ 13.6144901 , -7.67435074],
    [ 12.59641636, -9.37178184],
    [-19.25379655, 10.8531709 ],
    [ 12.59641636, -9.37178184],
    [ 13.05506301, -7.65279881],
    [ 13.05506301, -7.65279881],
])
```

### Expected Results

No warning (or a clearer one, e.g. "the dataset is degenerate and cannot be fitted with the current settings; please increase support_fraction").

### Actual Results

RuntimeWarning: Determinant has increased; this should not happen: log(det) > log(previous_det) (-6.080855240649472 > -46.216908559817156). You may want to try with a higher value of support_fraction (current value: 0.520).

### Versions

```shell
System:
    python: 3.9.6 | packaged by conda-forge | (default, Jul 11 2021, 03:39:48)  [GCC 9.3.0]
executable: /home/antony/miniconda3/envs/default/bin/python
   machine: Linux-5.15.15-200.fc35.x86_64-x86_64-with-glibc2.34

Python dependencies:
          pip: 21.3.1
   setuptools: 59.1.0
      sklearn: 1.0.2
        numpy: 1.22.1
        scipy: 1.7.3
       Cython: 0.29.24
       pandas: 1.3.2
   matplotlib: 3.6.0.dev1465+gf0593a6daf
       joblib: 1.0.1
threadpoolctl: 3.0.0

Built with OpenMP: True
```

user: anntzer
issue_22335: state: open
title: LinearDiscriminantAnalysis method fit_transform needs x and y
body: ### Describe the issue linked to the documentation

It is says that "fit_transform" from LinearDiscriminantAnalysis can be used as unsupervised transformations by omitting or setting y as None.

![image](https://user-images.githubusercontent.com/73899890/151680269-3bb4fe98-3b7b-4a7f-b9f1-2e7a6d10d1f3.png)

Which is not possible because we need a trained lda.

### Suggest a potential alternative/fix

change:
yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None
Target values (None for unsupervised transformations).

to:
yarray-like of shape (n_samples,)
Target values.
user: GaetanDu
labels: Needs Decision: Requires decision
issue_22330: state: open
title: ENH Add eigh as a solver in MDS
body: #### Reference Issues/PRs
Fixes #15272
Resolves #16067 (Stalled)

#### What does this implement/fix? Explain your changes.
PR #16067:
Adds implementation of Multidimensional scaling (MDS), which uses Singular Value Decompostion (SVD) method. SVD works much faster and far more accurate for Euclidean matrixes than SMACOF algorithm (current implementation of MDS in sklearn/manifold module).

This PR:
Addresses review comments and provides further forum for discussion regarding API.


#### Any other comments?
user: Micky774
issue_22310: state: open
title: Random Windows wheel build error: failed with exit status 1158
body: The nightly windows builds randomly fail with the following error:

```
error: Command "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\cibw\python\python.3.10.1\tools\libs /LIBPATH:C:\cibw\python\python.3.10.1\tools\PCbuild\amd64 /LIBPATH:build\temp.win-amd64-3.10 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\lib\x64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\um\x64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\lib\x64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\um\x64 /EXPORT:PyInit__hierarchical_fast build\temp.win-amd64-3.10\Release\sklearn\cluster\_hierarchical_fast.obj /OUT:build\lib.win-amd64-3.10\sklearn\cluster\_hierarchical_fast.cp310-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.10\Release\sklearn\cluster\_hierarchical_fast.cp310-win_amd64.lib /openmp" failed with exit status 1158
```

https://github.com/scikit-learn/scikit-learn/runs/4961291550?check_suite_focus=true

The above happened with `cp310-win_amd64` but apparently this can happen with other Python versions. Generally restarting the build is enough to get it to work again.

Let's try to collect below which config and file fail, to see if there is a pattern:

- `cp310-win_amd64` / `sklearn\cluster\_hierarchical_fast.cp310-win_amd64.lib`
- `cp38-win_amd64` / `sklearn\ensemble\_gradient_boosting.cp38-win_amd64.lib`
- `cp38-win32` / `sklearn\ensemble\_hist_gradient_boosting\_binning.cp38-win32.lib`
- `cp38-win32` / `sklearn\utils\_openmp_helpers.cp38-win32.lib`
- `cp39-win32`/ `sklearn\decomposition\_cdnmf_fast.cp39-win32.lib` 

[Link to all wheel building runs that were scheduled](https://github.com/scikit-learn/scikit-learn/actions/workflows/wheels.yml?query=branch%3Amain+event%3Aschedule)

user: ogrisel
issue_22285: state: open
title: ENH Support 2d y_score with 2 classes in top_k_accuracy_score w/ labels
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Related to #21173

#### What does this implement/fix? Explain your changes.
If `labels` are passed in, I think it safe to consider the problem multiclass.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
issue_22283: state: open
title: BUG? LinearSVC with hinge loss and L2 penalty, solver='liblinear' seems to get stuck compared to other solvers
body: ### Describe the bug

Compared to a vanilla coordinate descent on the dual, liblinear plateaus at a higher objective value.

### Steps/Code to Reproduce
```python
import warnings
import numpy as np
import matplotlib.pyplot as plt
from numba import njit
from numpy.linalg import norm
from sklearn.svm import LinearSVC

from sklearn.exceptions import ConvergenceWarning
warnings.filterwarnings('ignore', category=ConvergenceWarning)

C = 0.1
clf = LinearSVC(C=C, penalty='l2', dual=True, fit_intercept=False, tol=1e-12,
                loss='hinge', random_state=0)


@njit
def loss(w, yX, C):
    return np.maximum(0, 1 - yX @ w).sum() + norm(w) ** 2 / (2 * C)


@njit
def cd_dual(yX, C, n_iter):
    """Solve problem with coordinate descent in the dual, ie minimize:
    norm(yX @ mu) ** 2 / 2 + mu.sum()    subject to 0 <= mu <= C

    with link equation w = yX @ mu  (=sum_i mu_i * y_i * X[i]"""
    n_samples, n_features = yX.shape
    lipschitz = np.zeros(n_samples)
    for j in range(n_samples):
        lipschitz[j] = norm(yX[j]) ** 2

    mu = np.zeros(n_samples)
    w = np.zeros(n_features)
    losses = np.zeros(n_iter)
    for it in range(n_iter):
        for j in range(n_samples):
            old_mu_j = mu[j]
            grad_f_j = yX[j] @ w - 1
            new_mu_j = max(0, min(old_mu_j - grad_f_j / lipschitz[j], C))

            if new_mu_j != old_mu_j:
                w += yX[j] * (new_mu_j - old_mu_j)
                mu[j] = new_mu_j
        losses[it] = loss(w, yX, C)

    return w, losses


fig, axarr = plt.subplots(1, 2, figsize=(10, 4), constrained_layout=True)
for idx, (n_samples, n_features) in enumerate([[100, 500], [500, 100]]):
    np.random.seed(0)
    X = np.random.randn(n_samples, n_features)
    y = X @ np.random.randn(n_features) + 0.1 * np.random.randn(n_samples)
    y = 2 * (y > 0) - 1
    yX = X * y[:, None]

    max_iter = 3000 if n_samples > n_features else 300
    losses_sk = []
    iter_sk = np.geomspace(1, max_iter, num=50).astype(int)
    for n_iter in iter_sk:
        clf.max_iter = n_iter
        clf.fit(X, y)
        w = clf.coef_[0]
        losses_sk.append(loss(w, yX, C))
    losses_sk = np.array(losses_sk)

    w, losses_cd = cd_dual(yX, C, max_iter)
    loss_min = min(losses_cd.min(), losses_sk.min())

    axarr[idx].semilogy(iter_sk, losses_sk - loss_min, label='sklearn')
    axarr[idx].semilogy(losses_cd - loss_min, label='dual coordinate descent')
    axarr[idx].set_xlabel("iteration")
    axarr[idx].set_ylabel("loss - min loss")
    axarr[idx].set_title(f'n_samples, n_features=({n_samples}, {n_features})')
    axarr[idx].legend()
plt.show(block=False)


# no intercept is fitted
np.testing.assert_allclose(clf.decision_function(X), X @ clf.coef_[0])
```

### Expected Results

I expect both orange and blue suboptimality curve to go to zero. The blue one (sklearn) does not. This is particularly visible when n_samples > features (right plot)

### Actual Results

![image](https://user-images.githubusercontent.com/8993218/150823358-ae25d607-1a0c-4685-95dc-6acb134e43b0.png)


### Versions

```shell
System:
    python: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0]
executable: /home/mathurin/anaconda3/bin/python
   machine: Linux-5.13.0-27-generic-x86_64-with-glibc2.31

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 1.0.1
        numpy: 1.20.3
        scipy: 1.7.1
       Cython: 0.29.24
       pandas: 1.3.4
   matplotlib: 3.4.3
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True
```

user: mathurinm
issue_22262: state: open
title: Failing test test_quantile_estimates_calibration with q=.9
body: ### Describe the bug

Hello,

I built the source of the main branch and ran the tests. The quantile estimates calibration test is failing after 2 mins 33 sec and a lot of processor usage. 

Thanks!

### Steps/Code to Reproduce

I launched the test in my dev folder with:

```
pytest -v -k test_quantile_estimates_calibration
```

### Expected Results

Test passed

### Actual Results

```python-traceback
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.10.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /home/francois/mambaforge/envs/sklearn-env/bin/python
cachedir: .pytest_cache
rootdir: /home/francois/Documents/scikit-learn, configfile: setup.cfg, testpaths: sklearn
plugins: xdist-2.5.0, forked-1.4.0
collected 26675 items / 26672 deselected / 3 selected                                                                                                                                                     

sklearn/linear_model/tests/test_quantile.py::test_quantile_estimates_calibration[0.5] PASSED                                                                                                        [ 33%]
sklearn/linear_model/tests/test_quantile.py::test_quantile_estimates_calibration[0.9] FAILED                                                                                                        [ 66%]
sklearn/linear_model/tests/test_quantile.py::test_quantile_estimates_calibration[0.05] PASSED                                                                                                       [100%]

================================================================================================ FAILURES =================================================================================================
________________________________________________________________________________ test_quantile_estimates_calibration[0.9] _________________________________________________________________________________

q = 0.9

    @pytest.mark.parametrize("q", [0.5, 0.9, 0.05])
    def test_quantile_estimates_calibration(q):
        # Test that model estimates percentage of points below the prediction
        X, y = make_regression(n_samples=1000, n_features=20, random_state=0, noise=1.0)
        quant = QuantileRegressor(
            quantile=q,
            alpha=0,
            solver_options={"lstsq": False},
        ).fit(X, y)
>       assert np.mean(y < quant.predict(X)) == approx(q, abs=1e-2)
E       assert 0.721 == 0.9 ± 1.0e-02
E         +0.721
E         -0.9 ± 1.0e-02

sklearn/linear_model/tests/test_quantile.py:129: AssertionError
================================================================ 1 failed, 2 passed, 26672 deselected, 1032 warnings in 153.11s (0:02:33) =================================================================

```

### Versions

```
System:
    python: 3.10.2 | packaged by conda-forge | (main, Jan 14 2022, 08:02:09) [GCC 9.4.0]
executable: /home/francois/mambaforge/envs/sklearn-env/bin/python
   machine: Linux-5.13.0-27-generic-x86_64-with-glibc2.31

Python dependencies:
          pip: 21.3.1
   setuptools: 59.8.0
      sklearn: 1.1.dev0
        numpy: 1.22.0
        scipy: 1.7.3
       Cython: 0.29.26
       pandas: 1.3.5
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.0.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/francois/mambaforge/envs/sklearn-env/lib/libopenblasp-r0.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 8

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/francois/mambaforge/envs/sklearn-env/lib/libgomp.so.1.0.0
        version: None
    num_threads: 8
```
user: francoisgoupil
issue_22233: state: open
title: [WIP] Brier score binless decomposition
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Closes #21774. See also #18268, #21718


#### What does this implement/fix? Explain your changes.
Described in #21774


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: ColdTeapot273K
issue_22232: state: open
title: Sphinx docs don't build correctly on Windows 10
body: ### Describe the bug

I'm on Windows 10. The scikit-learn contributor docs prescribe the following to build the docs (after necessary installs): 
![image](https://user-images.githubusercontent.com/34613774/149794271-b07390d7-5c60-4c21-b7c5-3f6a69565222.png)

However, in practice, running `make` only results in: 
![image](https://user-images.githubusercontent.com/34613774/149794451-91c7aa44-2020-46eb-8dca-fbc87b50ea93.png)

Then, I attempt to run `make html-noplot`

### Steps/Code to Reproduce

```
cd doc
make
make html-noplot
```

### Expected Results

No error is thrown and docs are properly built with no plots.

### Actual Results

![image](https://user-images.githubusercontent.com/34613774/149795038-dc2a2ee9-be86-452a-89fb-9b8ee321a922.png)
![image](https://user-images.githubusercontent.com/34613774/149795120-447955de-2b2b-4daa-a7cc-543b7e8d357a.png)
this continues for a while, until
![image](https://user-images.githubusercontent.com/34613774/149795516-9b51a639-af59-4e65-b5c1-20f71379f7de.png)
which continues for a while, until it crashes at the end
![image](https://user-images.githubusercontent.com/34613774/149795707-b0e696e1-4a73-4ec1-a26e-5828427ed08a.png)

Finally, checking the actual directory I find only a `module/generated` directory with several images inside.

### Versions

![image](https://user-images.githubusercontent.com/34613774/149796023-401770e6-3485-43fd-b44f-df4e0ff2c0fe.png)

user: Micky774
issue_22227: state: open
title: [MRG] [ENH] Adds a `return_std_of_f` flag for GPCs
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #22226. 


#### What does this implement/fix? Explain your changes.
Adds a `return_std_of_f` flag for binary Gaussian Process Classifiers. This value was already being computed in some of the methods, so this PR just exposes that variable and adds checks in the API to ensure this is only possible for binary (and not multiclass) classification using the Laplace approximation.

#### Any other comments?

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: miguelgondu
labels: Needs Decision - API: Requires decision regarding API
issue_22226: state: open
title: Add a `return_std_of_f` kwarg to GPC's `predict` and `predict_proba`, just like the one GPR has
body: ### Describe the workflow you want to enable

Gaussian Process Regression in sklearn comes with the affordance to return standard deviations of the predictions in the `gpr.predict` method. The Gaussian Process Classifier doesn't. The main reason why (I assume) is that we don't have direct access to the standard deviation of our class estimates. We do, however, have access to the standard deviation of the *latent variable* f, and giving users access to it may allow them to have access to uncertainty estimates of the predicted classes by e.g. ancestral sampling or other Monte Carlo methods.

In this issue, I propose we add a keyword argument `return_std_of_f=False` to the methods `predict` and `predict_proba` of `sklearn.gaussian_processes.GaussianProcessClassifier`. This would allow us to return the standard deviation of the latent function f, a value that is already being computed in the `predict_proba` method.

### Describe your proposed solution

The solution is actually pretty simple, since the computations are already being done in the method `gpc.predict_proba`. Namely, [This line](https://github.com/scikit-learn/scikit-learn/blob/1d1aadd0711b87d2a11c80aad15df6f8cf156712/sklearn/gaussian_process/_gpc.py#L310) corresponds to computing the variance of the latent function $f$'s prediction on the test points. This addition would only need to be

1. The same computation under an `if` for the `gpc.predict` method.
2. An `if` statement with different returns according to the `return_std_of_f` flag for the `gpc.predict_proba` method.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: miguelgondu
labels: Needs Decision - API: Requires decision regarding API
issue_22196: state: open
title: Error in TFIDF vectorizer in "char_wb" analyzer
body: ### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/22195

<div type='discussions-op-text'>

<sup>Originally posted by **Pruthwik** January 12, 2022</sup>
For Whitespace sensitive char-n-gram tokenization, TFIDF vectorizer provides "char_wb" analyzer. But in the _char_wb_ngrams function, the word is padded with spaces at both ends. In the implementation, it introduces a space after character ngrams. I will explain this using a code snippet:

```python
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(analyzer='char_wb')
text = ['sachin is a good player .', 'he is a good man .']
X = tfidf.fit_transform(text)
print(tfidf.get_feature_names_out())
```

output is :

```
array([' ', '.', 'a', 'c', 'd', 'e', 'g', 'h', 'i', 'l', 'm', 'n', 'o', 'p', 'r', 's', 'y'], dtype=object)
```

the space should not be there as the first element. It is more evident if max_n > 1; if the same text is passed through 

```python
tfidf = TfidfVectorizer(analyzer='char_wb', ngram_range=(1, 2))
X = tfidf.fit_transform(text)
print(tfidf.get_feature_names_out())
```

the output generated is:

```
array([' ', ' .', ' a', ' g', ' h', ' i', ' m', ' p', ' s', '.', '. ','a', 'a ', 'ac', 'an', 'ay', 'c', 'ch', 'd', 'd ', 'e', 'e ', 'er',
'g', 'go', 'h', 'he', 'hi', 'i', 'in', 'is', 'l', 'la', 'm', 'ma',
'n', 'n ', 'o', 'od', 'oo', 'p', 'pl', 'r', 'r ', 's', 's ', 'sa', 'y', 'ye'], dtype=object)
' p', 's '
```

should not be in the output.
----- This is my observation, can the sklearn developer clarify. If we only left pad space for a word in the _char_wb_ngrams function, then this issue of generating ngrams with space is avoided.</div>
user: Pruthwik
labels: Needs Investigation: Issue requires investigation
issue_22178: state: open
title: power_t: does it make sense for this parameter to have negative values
body: sklearn/linear_model/_stochastic_gradient.py

Ref #22115

>I do not anticipate negative `power_t` to be mathematically meaningful but apparently our code accepts it without crashing... So ok with documenting it.

_Originally posted by @ogrisel in https://github.com/scikit-learn/scikit-learn/pull/22115#r780109928_

@thomasjpfan 
>I feel like this a case where documenting `-inf` will lead to more people trying out. If this is not mathematically meaningful, then we could be promoting a bad practice?

cc:  @glemaitre
user: reshamas
issue_22173: state: open
title: Improve HGBT leaf values updates
body: ### Describe the workflow you want to enable

This issue references this `TODO` comment:

https://github.com/scikit-learn/scikit-learn/blob/be89eb75f250dc5a769281939ba01e570fb12ae1/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py#L65-L67
### Describe your proposed solution

Some APIs (like [`std::nth_element`](https://en.cppreference.com/w/cpp/algorithm/nth_element)) allows getting the median or a given quantile of a contiguous buffer in O(n) but it does need to mutate a data-structure to sort (it can be the buffer or another data-structure if using a Comparator).

Could it be used there?

Moreover, why can't we compute the median in parallel here?

### Additional context

Follows-up with discussions: https://github.com/scikit-learn/scikit-learn/pull/20811/files/4df17828e439ad09a7784e99cc3d8d956eb50fe0#r781036357

---

/cc @NicolasHug who might be interested in following this issue as he initially worked on the update and authored this comment.
user: jjerphan
issue_22172: state: open
title: Add positive / nonnegative constraint to ARDRegression
body: ### Describe the workflow you want to enable

The ARDRegression is phenomenal. Would it be possible to enforce nonnegative constraints in this function?

### Describe your proposed solution

adding a positive=True parameter similar to LinearRegression. Of course I am not sure if the method itself allows for such a constrain. 

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: daliagachc
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_22170: state: open
title: Partial_fit for ColumnTransformer
body: ### Describe the workflow you want to enable

ColumnTransformer does not support partial_fit() even if individual transformers do (like StandardScaler). Is it possible to add a partial_fit routine?

### Describe your proposed solution

1. Don't reset state
2. Add _partial_fit to pipeline.py. something like:
```
def _partial_fit_one(transformer, X, y, weight, message_clsname="", message=None, **fit_params):
    """
    Fits ``transformer`` to ``X`` and ``y``.
    """
    with _print_elapsed_time(message_clsname, message):
        return transformer.partial_fit(X, y, **fit_params)
```
if the transformer hasattr partial_fit

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: Krasner
labels: Needs Decision - API: Requires decision regarding API
issue_22117: state: open
title: Efficient clustering (unknown number of clusters; optionally online) by greedily picking exemplars
body: ### Describe the workflow you want to enable

Cluster a dataset (many samples) where **distances are small within clusters and large between clusters**. **Memory requirements and runtime** should be low.

### Describe your proposed solution

If a sample's distance to all exemplars (that were defined so far) is above a user-defined threshold, concatenate that sample to the exemplars. Continue with the next samples.

The runtime requirements seem to be in O(`n_samples*n_clusters`), and memory requirements in O(`n_clusters`).

In practice this seemed to be by far the fastest and most memory-economic clustering algorithm for my dataset (as said: **many samples, well separated clusters**).

Here's the code:

```python
DISTANCE_THRESHOLD = 100

# Initialize array of exemplars using the first sample, in order to avoid cheking in each loop iteration whether the exemplar array is empty.
exemplars = samples[0:1,:]

for batch in batches:
    # If one of the new samples is far away from all exemplars, concatenate it to the exemplars.
    while (distances_to_nearest_exemplars := (distance_matrix := scipy.spatial.distance.cdist(batch, exemplars)).min(axis=1)).max() > DISTANCE_THRESHOLD:
        new_exemplar_index = distances_to_nearest_exemplars.argmax()
        exemplars = np.concatenate([exemplars, batch[new_exemplar_index:new_exemplar_index+1,:]])
        # TODO check whether keeping exemplars in a list rather than a Numpy array is faster.
        # `distance_matrix.argmax(axis=1)` can be used to compute cluster assignments for the samples.
        # Deleting the sample from the batch is optional because the sample will be below the threshold after recomputing the ditances
        #   (which the `while` loop recomputes anyway in order to avoid adding several similar samples from the same batch).
        # `scipy.spatial.distance.cdist` seems faster than `scipy.spatial.distance_matrix`.
```

Optionally, **exemplars (centroids) can be updated whenever desired** by averaging the samples assigned to their cluster so far.


### Describe alternatives you've considered, if relevant

I tried various [clustering algorithms from sklearn](https://scikit-learn.org/stable/modules/clustering.html#overview-of-clustering-methods) for **(very) large `n_samples`**, but they were too memory-hungry and slow.

### Additional context

As a bonus, the algorithm is applicable to **online** clustering.

Knowing the number of clusters is **not necessary**.

If a specific number of clusters is desired, this can be achieved as described below.

Centroids can be updated in the end (see above) and iteratively merged, if a cluster **hierarchy** (or a smaller number of clusters) is desired. (To get a larger number of clusters first, the threshold can be set low.)
user: page200
labels: Needs Decision: Requires decision
issue_22095: state: open
title: Updating already built KDTree with another KDTree
body: ### Describe the workflow you want to enable

Hi,

I am building KDTree with very large and dense point clouds. Once I am building these trees, I am dumping them in temporary storage location so that I don't waste time by building the tree again.

But the issue is - I need combination of 5 to 7 kd-trees now. Is there any way using which I can combine the already build kd-trees into single kd-tree?

### Describe your proposed solution

Update method for KDTree class

### Describe alternatives you've considered, if relevant

Na

### Additional context

Na
user: INF800
issue_22078: state: open
title: first attempt to change iterative_imputer
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: siavrez
issue_22046: state: open
title: ENH Add Multiclass Brier Score Loss
body: Resolves #16055.
This PR updates #18699 by @aggvarun01 after a merge with `main` and resolves merge conflicts. I do not have the permissions to push directly in the original branch and opening a sub-PR pointing to #18699 would lead to an unreadable diff because of the one-year merge sync.

I also added a changelog entry and demonstrate the new function in the multiclass calibration example.

@aggvarun01 if you want feel free to pull the last commit from this commit from this branch to your branch. Alternatively we can finalize the review here.
user: ogrisel
issue_22044: state: open
title: Add a way to use `warm_start` together with `cross_val_score`?
body: ### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/22042

<div type='discussions-op-text'>

<sup>Originally posted by **PGijsbers** December 21, 2021</sup>
I want to obtain scores for a random forest with a different number of estimators across the same fold in a reproducible manner. I essentially want something like:

```python
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

x, y = make_classification(random_state=0)
rf = RandomForestClassifier(n_estimators=10, warm_start=True, random_state=0)
step_size = 10

for i in range(3):
    scores = cross_val_score(rf, x, y, cv=3)
    print(scores)
    rf.n_estimators += step_size
```

However, from my understanding each time the forests are trained from scratch in this scenario, since the original `rf` object is cloned. Instead I would want to save the progress for each fold and use that in the next iteration. It is possible to get the fitted estimators with `cross_validate(..., return_estimator=True)` but I don't see any compatible method to pass an estimator for each fold.
 Is there a way? Or should I instead open an issue for a feature request? :)</div>
 
 ---
 
 Suggested change: allow `cross_validate`'s estimator parameter to optionally be a list of estimators which matches in length to the number of folds. Then from the user it could work something similar to:
 
 ```python
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_validate

x, y = make_classification(random_state=0)
rf = RandomForestClassifier(n_estimators=10, warm_start=True, random_state=0)
k_folds = 3
forests = [rf] * k_folds
step_size = 10

for i in range(3):
    result = cross_validate(forests, x, y, cv=k_folds, return_estimator=True)
    forests = result["estimator"]
    for forest in forests:
        forest.n_estimators += step_size
 ```
 
 Internally you can presumably just `zip` the estimators in [here](https://github.com/scikit-learn/scikit-learn/blob/0d378913be6d7e485b792ea36e9268be31ed52d0/sklearn/model_selection/_validation.py#L283) and be done as I assume that the returned `estimator` list is already in matching order with the folds.
user: PGijsbers
issue_22043: state: open
title: FEA add pinball loss to SGDRegressor
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Partially addresses: #20132
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
Added the pinball loss to sgd_fast.pyx.


#### Any other comments?
The results doesn't seems to be comparable with the QuantileRegressor. Need help.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: venkyyuvy
issue_22041: state: open
title: Using a RandomForest's `warm_start` together with `random_state` is poorly documented
body: ### Describe the issue linked to the documentation

Consider the following example:
```
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier

x, y = make_classification(random_state=0)
rf = RandomForestClassifier(n_estimators=1, warm_start=True, random_state=0)

rf.fit(x, y)
rf.n_estimators += 1
rf.fit(x, y)
```
According to [controlling randomness](https://scikit-learn.org/stable/common_pitfalls.html#randomness), when `random_state` is set:
> If an integer is passed, calling fit or split multiple times always yields the same results.

But calling `fit` multiple times in a warm start setting does not yield the same results (as expected, we want more trees, and we want different trees). The example above produces a forest with two unique trees, and the overall forest is identical to creating at once with `RandomForestClassifier(n_estimators=2, warm_start=False, random_state=0)`. The same behavior is observed when a `numpy.random.RandomState` is used.

However, I found it (at first) impossible to determine this behavior from the documentation alone. As far as I am aware, the only hint that should have helped me is this [warm_start](https://scikit-learn.org/stable/glossary.html?highlight=warm%20start%20cross%20validation#term-warm_start) documentation:
>  When warm_start is true, the existing fitted model attributes are used to initialize the new model in a subsequent call to fit.

In hindsight, the internal `random state`-object likely counts as a "fitted model attribute" which would allow you to infer the behavior from the documentation.

### Suggest a potential alternative/fix

I am not sure if this behavior is consistent across all estimators which support the `warm_start` parameter. A clarification in the `warm_start` section makes the most sense to me. Either a single sentence or a small paragraph depending on whether or not there are differences between the different estimators.

---

I'd be willing to set up the PR but I figure it makes sense to agree on the action (if any) and wording first.
user: PGijsbers
issue_22039: state: open
title: WIP Ensemble Classifiers handles class weights
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
This is an alternative to PR #19753 . Instead of only working on `VotingClassifier`, this PR fixes the following ensemble classifiers:
- VotingClassifier
- BaggingClassifier
- StackingClassifier
- AdaBoostClassfier

#### What does this implement/fix? Explain your changes.
For **BaseEnsemble**:
Added another argument `encode_class_weight` indicating if the sub-estimator's `class_weight` need to be encoded. Since we only need to encode class weight for bagging classifiers (not boosting classifiers)

For **_BaseHeterogeneousEnsemble**:
Encode `class_weight` for all the estimators.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: MaxwellLZH
issue_22022: state: open
title: [MRG] Add a separator between a criteria and other contents in a node in export_graphviz for visibility
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->
#### Reference Issues/PRs
Fixes #7774 . Add a separator line. 

<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.

Add a separator line between a criteria and other contents in node in `export_graphviz` which increase visibility of exported tree. Precisely the PR introduced a new boolean parameter `add_separator` to `export_graphviz` method. When `add_separator` is set as True, a separator line is added in an each node. This is achieved by using Record-based Nodes (see https://graphviz.org/doc/info/shapes.html#record). 

#### Any other comments?

A node shape becomes `record` not `box` with `add_separator=True`. This PR does not break backward compatibility because  `add_separator=False` in default.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->
Obtained graphs are shown below.
`add_separator=False` (conventional)
![old](https://user-images.githubusercontent.com/16839037/147192755-289db8e2-2932-4d93-a628-9df78eb05e38.png)
`add_separator=True` (new)
![new](https://user-images.githubusercontent.com/16839037/147192767-22d64614-80c9-4482-acf9-986c5485dc7e.png)
`"class_names":True,
          "filled":True,
          "leaves_parallel":True,
          "node_ids":True,
          "proportion":True,
          "rotate":True,
          "rounded":True,
"add_separator":True,
"special_characters":True}
` (separator with other parameters activated)
![all](https://user-images.githubusercontent.com/16839037/147192840-0768fb4d-74b1-41cb-873e-f97f69a1f29d.png)


user: takoika
issue_22021: state: open
title: FIX Add BaggingClassifier support for class_weights with string labels
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #19665

#### What does this implement/fix? Explain your changes.
This PR enables `BaggingClassifier` to re-encode the `class_weights` for `base_estimator`.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
issue_22010: state: open
title: [WIP] Make it possible to pass an arbitrary classifier as method for CalibratedClassifierCV
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #21280 

#### What does this implement/fix? Explain your changes.
- [x] Implement a third calibration method using a custom classifier in CalibratedClassifierCV.
- [x] Update existing tests with this new method.
- [ ] Create new tests specific to this method.
- [ ] Extend at least one calibration example to demonstrate this new capability.
- [ ] Create a new example highlighting the advantages of this method compared to `sigmoid` and `isotonic`.
- [ ] Update the [Probability calibration](https://scikit-learn.org/stable/modules/calibration.html) page in the user guide with a paragraph on this method. 

#### Any other comments?
This PR replaces #21992 as issue #21280 was updated.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: aperezlebel
issue_22008: state: open
title: fix skip unnecessary check of gram_matrix in coordinate_descent
body: Example: Fixes #21997. See also #19004.

user: derHeinzer
labels: Needs Reproducible Code: Issue requires reproducible code
issue_22004: state: open
title: ENH column_or_1d cast pandas boolean extension to bool ndarray
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes https://github.com/scikit-learn/scikit-learn/issues/17675
Closes https://github.com/scikit-learn/scikit-learn/pull/17764 (Alternative)


#### What does this implement/fix? Explain your changes.
When calling `check_X_y`, `y` ends up in `column_or_1d`. This PR checks for the Boolean dtype directly and cast that into a bool.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
issue_22000: state: open
title: [WIP] Callback API continued
body: Fixes #78 #7574 #10973
Continuation of the work started in #16925 by @rth.

### Goal
The goal of this PR is to propose a callback API that can handle the most important / asked use cases.
- **Convergence monitoring** (#14338 https://github.com/scikit-learn/scikit-learn/issues/8994#issuecomment-306772460)
  Monitor some quantities / metrics at each iteration.
  This can also be very useful for maintenance / debugging / implementation of new features.
- **Early stopping** (#10973)
  Allow to stop the iterations based on some external metric evaluated on a validation set.
- **Progress bars** (#7574 #78 #10973)
- **Logging / Verbose** (#78)
- **Snapshots**
  Take regular snapshots of an estimator during the fit to be able to recover a working estimator if the fit is somehow interrupted for instance.
- **Resource monitoring**
  (not implemented yet)

### Challenges
- Supporting all these features and make each of these callbacks available is not easy and will require some refactoring in probably many estimators.
  
  The proposed API makes it possible to enable the callbacks 1 estimator at a time: Setting callbacks on non yet supported estimators has no effect. Thus we can then incrementally do it in subsequent dedicated PRs. Here I only did NMF, LogisticRegression and Pipeline to show what are the necessary changes in the code base.
  
  The proposed API also makes it possible to only enable a subset of the features for an estimator, and add the remaining ones later. For LogisticRegression I only passed the minimum for instance.

- Callbacks should not impact the performance of the estimators. Some quantities passed to the callbacks might be costly to compute. We don't want to spend time computing them if the only callback is a progress bar for instance.

  The solution I found is to do a lazy evaluation using lambdas and only actually compute them if there's at least 1 callback requesting it. For now callbacks can request these by defining specific class attributes but maybe there's a better way. mixins ?

- The callbacks described above are not meant to be evaluated a the same fitting step of an estimator.

  When an estimator has several nested loops (LogisticRegressionCV(multiclass="ovr") for instance has a loop over Cs, a loop over the classes and then the final loop for the iterations on the dataset), the snapshot callback can only be evaluated at the end of an outermost loop while the EarlyStopping would be evaluated at the end of an innermost loop, and the ProgressBar could be evaluated at each level of nesting.

  In this PR I propose that each estimator holds a `computation tree` as a private attribute representing these nested loops, the root being the beginning of fit and each node being one step of a loop. This structure is defined in `_computation_tree.py`. It allows to have a simple way to know exactly at which step of the fit we are at each evaluation of the callbacks and is kind of the best solution I found to solve the challenges described below. This imposes the main changes to the code base, i.e. passing the parent node around.

- Dealing with parallelism and especially multiprocessing is the main challenge to me.

  Typically with a callback you might want to accumulate a bunch of info during fit and recover them at the end. The issue is that the callback is not shared between sub-processes and modifying its state in a sub-process (e.g. modifying an attribute) will not be visible from the main process. The joblib API doesn't allow inter-process communication that would be needed to overcome this.

  The solution we found is that the callbacks write the information they want to keep in files (in files in this first implementation but we might consider sockets or another solution ?). It's relatively easy to avoid race conditions with this design.
  As an example this is necessary to be able to report progress in real time. In an estimator running in parallel, there's no like current computation node. We are at different nodes at the same time. But having the status of each node in a file updated at each call to the callbacks allows to know the current overall progress from the main process. (there are other difficulties described later).

- The last main challenge is meta-estimators. We'd like some callbacks to be set on the meta-estimator, like progress bars, but some others to be set on the underlying estimator(s), like early-stopping. Moreover we encounter the parallelism issue again if the meta-estimator is fitting clones of the underlying estimator in parallel, like GS.

  For that, I propose to have a mixin to tell a callback that it should be propagated to sub estimators. This way the meta-estimator will only propagate the appropriate callbacks to its sub-estimators, and these sub-estimators can also have normal callbacks.

### The API
This PR adds a new module ``sklearn.callback`` which exposes `BaseCallback`, the abstract base class for the callbacks. All callbacks must inherit from BaseCallback.  It also exposes `AutoPropagatedMixin`. Callbacks that should be propagated to sub-estimators by meta-estimators must inherit from this.

`BaseCallback` has 3 abstract methods:
- `on_fit_end`. Called at the beginning of fit, after all validations. We pass a reference to the estimator, X_train and y_train.
- `on_fit_iter_end`. Called at the end of each node of the computation tree, i.e. each step of each nested loop. We pass a reference to the estimator (which at this point might be different from the one passed at in_fit_begin for propagated callbacks), and the computation node where it was called. We also pass some of these:
  - stopping_criterion: when the estimator has a stopping criterion such that the iterations stop when `stopping_criterion <= tol`.
  - tol: tolerance for the stopping criterion.
  - reconstruction_attributes. These are the necessary attributes to construct an estimator (by copying the estimator and setting these as attributes) which will behave as if the fit stopped at this node. Then we must be able to call `predict`, `transform`, ...
  - fit_state: Model specific quantities updated during fit. This is not meant to be used by generic callbacks but by a callback designed for a specific estimator instead. This arg is not used in any of the use cases described above but I thinkit's important to have for custom callbacks. It's the role of each estimator to decide what is interesting to pass to the callback. We could later think of new field in the docstring of the estimators to describe what keys they pass to this arg.
- `on_fit_end`. Called at the end of fit. Takes no argument. It allows the callback to do some clean-up.

### Examples
- **Progress bars**. 
  
  <details><summary>expand</summary>

  Here's an example of progress monitoring using [rich](https://github.com/willmcgugan/rich). I used custom estimators to simulate a complex setting with a meta-estimator (like a GridSearchCV) running in parallel with a sub-estimator also running in parallel.
 
  https://user-images.githubusercontent.com/34657725/146427556-c8add1dd-55e1-43e8-bf0a-eed2ff3edefe.mp4

  </details>

- **Convergence Monitoring**

  <details><summary>expand</summary>

  ```py
  from sklearn.decomposition import NMF
  import numpy as np
  X = np.random.random_sample((1100, 100))
  X_val = X[-100:]
  nmf = NMF(n_components=20, solver="mu")
  callback = ConvergenceMonitor(X_val=X_val)
  nmf._set_callbacks(callback)
  nmf.fit(X[:1000])
  callback.plot()
  ```
  ![callback_ex1](https://user-images.githubusercontent.com/34657725/146428396-57f90f10-0805-40e2-ab34-097d5ec55f16.png)

  </details>

- **Snapshot**

  <details><summary>expand</summary>

  ```py
  from sklearn.decomposition import NMF
  import numpy as np
  X = np.random.random_sample((1100, 100))
  nmf = NMF(n_components=20, solver="mu")
  callback = Snapshot()
  nmf._set_callbacks(callback)
  nmf.fit(X[:1000])
  # interrupt fit. Ctrl-C for instance
  # [...]
  KeyboardInterrupt:

  import pickle
  with open(callback.directory / "2021-12-16_19-33-15-083014.pkl", "rb") as f:
      new_nmf = pickle.load(f)
  W = new_nmf.transform(X[-100:])
  ```

  </details>

- **EarlyStopping**
  
  <details><summary>expand</summary>

  If the `on_fit_iter_end` method of the callbacks returns True, the iteration loop breaks.
  ```py
  from sklearn.decomposition import NMF
  import numpy as np
  X = np.random.random_sample((1100, 100))
  X_val = X[-100:]
  nmf = NMF(n_components=20, solver="mu")
  callback = EarlyStopping(monitor="objective_function", X_val=X_val, max_no_improvement=10, tol=1e-4)
  nmf._set_callbacks(callback)
  nmf.fit(X[:1000])
  ```

  </details>

- **Verbose**

  <details><summary>expand</summary>

  ```py
  from sklearn.decomposition import NMF
  import numpy as np
  X = np.random.random_sample((1100, 100))
  nmf = NMF(n_components=20, solver="mu", max_iter=20)
  nmf._set_callbacks(TextVerbose())
  nmf.fit(X)
  ```
  ```
  [NMF] iter 0 | time 0.02493s | stopping_criterion=8.730E-01 | tol=1.000E-04
  [NMF] iter 1 | time 0.02634s | stopping_criterion=8.737E-01 | tol=1.000E-04
  [NMF] iter 2 | time 0.02768s | stopping_criterion=8.743E-01 | tol=1.000E-04
  [NMF] iter 3 | time 0.02893s | stopping_criterion=8.749E-01 | tol=1.000E-04
  [NMF] iter 4 | time 0.03016s | stopping_criterion=8.755E-01 | tol=1.000E-04
  [NMF] iter 5 | time 0.03136s | stopping_criterion=8.760E-01 | tol=1.000E-04
  [NMF] iter 6 | time 0.03255s | stopping_criterion=8.766E-01 | tol=1.000E-04
  [NMF] iter 7 | time 0.03375s | stopping_criterion=8.772E-01 | tol=1.000E-04
  [NMF] iter 8 | time 0.03496s | stopping_criterion=8.777E-01 | tol=1.000E-04
  [NMF] iter 9 | time 0.03691s | stopping_criterion=8.782E-01 | tol=1.000E-04
  [NMF] iter 10 | time 0.03841s | stopping_criterion=5.307E-04 | tol=1.000E-04
  [NMF] iter 11 | time 0.03966s | stopping_criterion=1.049E-03 | tol=1.000E-04
  [NMF] iter 12 | time 0.04087s | stopping_criterion=1.552E-03 | tol=1.000E-04
  [NMF] iter 13 | time 0.04209s | stopping_criterion=2.036E-03 | tol=1.000E-04
  [NMF] iter 14 | time 0.04327s | stopping_criterion=2.498E-03 | tol=1.000E-04
  [NMF] iter 15 | time 0.04447s | stopping_criterion=2.936E-03 | tol=1.000E-04
  [NMF] iter 16 | time 0.04565s | stopping_criterion=3.349E-03 | tol=1.000E-04
  [NMF] iter 17 | time 0.04686s | stopping_criterion=3.734E-03 | tol=1.000E-04
  [NMF] iter 18 | time 0.04804s | stopping_criterion=4.093E-03 | tol=1.000E-04
  [NMF] iter 19 | time 0.04923s | stopping_criterion=4.425E-03 | tol=1.000E-04 
  ```

  </details>

### TODO
This PR is still WIP. 
- [ ] It's missing all the documentation of the callback module to describe the API and how to use and write callbacks and an example.
- [ ] I started adding tests for the computation tree but we need more, and I still need to add test for the callback api and tests for each of the implemented callbacks.
- [ ] Finalize and Document the implemented callbacks. There are still a few issues that need to be fixed in these callbacks.
- [ ] Think about how callbacks should be reinitialized when reused, like refitting an estimator.
user: jeremiedbb
issue_21995: state: open
title: Showcase of new scikit-learn features and examples
body: ### Describe the issue linked to the documentation

The displaying pipelines example is buried currently in the documentation.

This example is buried:  
https://scikit-learn.org/dev/auto_examples/miscellaneous/plot_pipeline_display.html#sphx-glr-auto-examples-miscellaneous-plot-pipeline-display-py

### Suggest a potential alternative/fix

Is there a way to more prominently highlight new examples on the homepage?  
https://scikit-learn.org/dev/auto_examples/index.html

cc:  @thomasjpfan 
user: reshamas
labels: Hard: Hard level of difficulty
issue_21968: state: open
title: Add DateTimeTransformer?
body: ### Describe the workflow you want to enable

Do we want to add a transformer for pandas datetimes?
We haven't really added much based on pandas yet but this would be a pretty natural thing to add.
You could argue that you can do something like ``FunctionTransformer(lambda X: X.dt.dayofweek)`` or similar for the other features (year, hour, minute, month...) but the problem with that is that you don't get feature names, which is terrible for interpretation.

Featurizing datetimes is super common (the last ~10 datasets I worked on had it) and I think it's a workflow we should make easier.


### Describe your proposed solution

Implement a DateTimeTransformer that takes in maybe just a single column (that would work well with ColumnTransformer but is a bit different from other transformers, but quite similar to the CountVectorizer, so maybe if it takes a single column it should be ``DateTimeVectorizer``) and a list of features to derive, like ``dayofweek``, ``dayofyear`` etc, but which creates meaningful feature names.

### Describe alternatives you've considered, if relevant

An alternative would be to improve attaching feature names to ``FunctionTransformer``, but this would still require some non-trivial code for datetimes, and they are just very very common.
user: amueller
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_21967: state: open
title: Documenting missing-values practices
body: ## Describe the issue linked to the documentation

### Context

We discussed with @glemaitre and @GaelVaroquaux about documenting missing-values practices for prediction in scikit-learn as part of my PhD work at Inria (discussion [here](https://hackmd.io/LAxP9HduQMacG4Tf-p_j9Q?view)).

Indeed, the current documentation gives no recommendations on this point. We feel that there is now a better understanding of missing values in the context of supervised learning and thus that we have more hindsight on the theoretical and practical messages that would be helpful for the users than when the current documentation was written. We think it would be useful to restructure the documentation and examples to convey these messages.


### Messages to convey

#### Main messages:

1. Missing values in inference and in supervised learning are different problems with different tradeoffs. Define the terms and highlight the differences.
2. Don't impute jointly training and test sets: data leakage and can't use in production.
3. Simpler learners need powerful imputation (e.g conditional imputation with Iterative Imputer). Define conditional imputation. (theoretical arguments can be found in LeMorvan 2020).
4. Conditional imputation is guaranteed to work only for "ignorable missingness" (Missing At Random mechanism, to define). Otherwise, the mask is needed (missingness is seldom ignored: the data are missing for a reason). Wikipedia pages on missing values can justify this.
5. Powerful learners + simple imputation or no imputation works best (robustness to missingness mechanisms and flexibility), e.g HistGradientBoosting (this comes from experience, including systematic benchmarks).
6. For categorical features, impute missing values as a new category (imputing to an existing category destroys information important to the learner).
7. Computation cost of imputation can quickly get large, and even intractable for the most costly methods (e.g IterativeImputer, KNNImputer).

#### Side messages:
1. The optimal predictor on partially-observed values is not always "good" imputation + the optimal predictor on the fully-observed values (Le Morvan et al. 2021). You need to account for missingness in some way.
2. For multiple imputation, need to separate training and test behaviors (cf main message 1 above).
3. As a consequence, ensemble methods, such as bagging, are a good solution for implementing multiple imputation in practice (a single supervised learning applied to many imputations is likely severely suboptimal).

#### Take-home messages:
1. If little data: use conditional imputation and simple learners.
2. If a lot of data (n>1000), use HistGradientBoosting.
3. Don't impute categorical variables.

### Ressources

1. Wikipedia [Missing data](https://en.wikipedia.org/wiki/Missing_data)
1. Josse 2019 [On the consistency of supervised learning with missing values](https://arxiv.org/abs/1902.06931)
2. LeMorvan 2020 [Linear predictor on linearly-generated data with missing values: non consistency and solutions](http://proceedings.mlr.press/v108/morvan20a.html)
3. LeMorvan 2021 [What's a good imputation to predict with missing values?](https://arxiv.org/abs/2106.00311)
4. Perez-Lebel 2022 [Benchmarking missing-values approaches for predictive models on health databases](https://doi.org/10.1093/gigascience/giac013)

## Suggest a potential alternative/fix


After discussing with @glemaitre and @GaelVaroquaux, the following changes were suggested.

### Big picture

The goal is to give the recommendations above, and have simple examples that convey the right intuitions (even simple simulated data can be didactic by showing the basic mechanisms).

1. Write a narrative doc page that gives the big picture messages listed above and some figures.
2. Replace the current examples about imputation that don't give a clear message.
3. An example to generate simple figures explaining the difference between Missing At Random and Missing Not At Random (as in https://www.slideshare.net/GaelVaroquaux/dirty-data-science-machine-learning-on-noncurated-data/27).
   Didactic purpose: intuitions on the fact that missing values may distort distributions. Short example.
4. An example to develop intuitions on imputation with the interplay between imputation and learning, adapted from http://dirtydata.science/python/gen_notes/01_missing_values.html, but only the two first sections. Didactic purpose: How the mechanism + imputation modifies the link between X and y.
5. Adapt the docstrings to give local recommendations:
    a. IterativeImputer: give time complexity (algorithmic scalability) and say it is not a magic bullet in the face of structured missingness.
    b. KNNImputer: terrible computational scalability.
    c. SimpleImputer: does not work well with simple models.


### Proposed roadmap

(refers to above, and can be detailed in a board)

- [ ] 5.a #23701 
- [ ] 5.b
- [ ] 3 #23746
- [ ] 5.c #23714 
- [ ] 4
- [ ] 1
- [ ] 2
user: aperezlebel
issue_21962: state: open
title: Added a new method to specify monotonicity constraints using dict of feature names in HistGradientBoostingClassifier/Regressor. Fixes #21961
body: Added a new method to specify monotonicity constraints using dict of feature names in HistGradientBoostingClassifier/Regressor. 

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See  #https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #21961

#### What does this implement/fix? Explain your changes.
This implementation adds a method to specify the monotonicity constraints for HistGradientBoostingClassifier and Regressor.

Changes:
- sklearn/ensemble/_hist_gradient_boosting/grower.py
    - Added function to read a dict and generate an array-like representation of the constraints. 
    - Changed docstring to accommodate accepting dict as a valid input type.
    - Added validation checks to verify if input X is a dataframe type and has string columns
- sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py:
    - Added function to obtain column names for input X
    - Added parameter to pass the column names to grower.py
-  sklearn/ensemble/_hist_gradient_boosting/tests/test_monotonic_contraints.py:
    - Added tests
-  doc/modules/ensemble.rst:
    - Added documentation for accepting constraints as dict 


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Anirudhsekar96
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_21961: state: open
title: QoL improvement: Add monotonicity constraints specification using dict of feature names in HistGradientBoostingClassifier/Regressor
body: ### Describe the workflow you want to enable

As of current version, we need to specify monotonicity constraints for each feature as an array-like object ( e.g. [0,1,1] ) in HistGradientBoostingClassifier/Regressor. It would be a good quality of life improvement if we can specify the constraint as dict (e.g. {"feature_name1":0, "feature_name2":1 ...})  if the input X is a dataframe.

References:
- https://scikit-learn.org/stable/modules/ensemble.html#monotonic-cst-gbdt
- https://scikit-learn.org/stable/auto_examples/ensemble/plot_monotonic_constraints.html
- https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingClassifier.html#sklearn.ensemble.HistGradientBoostingClassifier

### Describe your proposed solution

Make the following changes inside fit function for HistGradientBoostingClassifier/Regressor:
- Change input type for monotonic_cst to array-like or dict.
- If constraint is specified as a dict, add a validation check for verifying if X is a dataframe with string column names.
- Iterate through column names in X, add constraint if specified in the dict to array-like object or add default value if no constraint is specified explicitly. Return array-like object in place of original dict.
 

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: Anirudhsekar96
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_21960: state: open
title: [WIP] decision_function for MultiOutputClassifier
body: #### Reference Issues/PRs
Fixes #21861

#### What does this implement/fix? Explain your changes.

This enhancement exposes the `decision_function` function of the estimator passed to MultiOutputClassifier. It allows users to leverage the decision/confidence scores if available for the estimator in the multi-class multi-output case. This follows closely the implementation of `predict_proba` which is exposed for estimators with this attribute. 


#### Any other comments?

I have also corrected the docstring of `predict_proba`  to mention the rising of an ``AttributeError`` instead of incorrectly a ``ValueError``.




user: chritter
issue_21942: state: open
title: Extend ClassifierChain to multi-output problems (ClassifierChain.decision_function)
body: **Reference Issues/PRs**
Fixes #13339
Fixes #9245

**What does this implement/fix? Explain your changes.**

I have done changes in "ClassifierChain.decision_function" method to support multioutput-multiclass.
Now the output will be (n_samples, n_classes, n_outputs) instead of (n_samples, n_classes), with an exception for binary classifier where n_classes will be one instead of two. 
I also have to make suitable changes in test function "test_classifier_chain_fit_and_predict_with_linear_svc" which is present in  "sklearn/tests/test_multioutput.py".

**Any other comments?**

This can break backward compatibility, since the output now of shape  (n_samples, n_classes, n_outputs).

user: rana-akkumar
issue_21935: state: open
title: ENH add x and y to importance getter rfe
body: #### Reference Issues/PRs
Fixes #21934 

#### What does this implement/fix? Explain your changes.
Adds possibility of passing also train instances and labels to importance_getter in Recursive Feature Elimination

user: ClaudioSalvatoreArcidiacono
issue_21934: state: open
title: Optionally passing X and y to importance_getter callable in Recursive Feature Elimination
body: ### Describe the workflow you want to enable

[Recursive Feature Importance](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html) has an optional initialisation parameter `importance_getter` can can be set to a callable. Accordingly to the documentation:

> If `callable`, overrides the default feature importance getter. The callable is passed with the fitted estimator and it should return importance for each feature.

It would be very nice to use SHAP values as a proxy for feature importance. This would enable using recursive feature elimination for models that do not support feature importance out of the box. In order to calculate SHAP feature importances, it could be possible to create an importance_getter that implements it. The current implementation does not allow to do so because, in order to Compute SHAP feature importances, also the training input samples and the target values are needed.

### Describe your proposed solution

I propose to pass to the importance_getter callable also the training input samples and the target values. In this way, we could combine RFE and shap values in the following way:

```python
import numpy as np
import shap
from sklearn.tree import DecisionTreeClassifier
from sklearn.feature_selection import RFE

def shap_importance_getter(clf, X, y):
    explainer = shap.Explainer(clf)
    shap_values = explainer.shap_values(X, y)[0]
    importance = np.abs(shap_values).mean(0)
    return importance

rfe = RFE(
    DecisionTreeClassifier(max_depth=3),
    n_features_to_select=3,
    importance_getter=shap_importance_getter
)

rfe.fit_transform(X, y)
```

### Describe alternatives you've considered, if relevant

There is already a package called [probatus](https://ing-bank.github.io/probatus/api/feature_elimination.html#probatus.feature_elimination.feature_elimination.ShapRFECV) that already provides a more ad-hoc implementation of RFE using shap values.


### Additional context

_No response_
user: ClaudioSalvatoreArcidiacono
issue_21932: state: open
title: categorical predictors in randomForest
body: ### Describe the workflow you want to enable

I would like to pass categorical predictors to the sklearn randomForestRegressor/Classifier without the need to dummy code them. 
Its potential improvement of predictive power is significant and this idea has been implemented in most competing randomForest packages in R (e.g. [ranger](https://cran.r-project.org/web/packages/ranger/index.html) or [randomForest](https://cran.r-project.org/web/packages/randomForest/index.html)) as well as the h20 library.

### Describe your proposed solution

Following the documentation in [ranger](https://cran.r-project.org/web/packages/ranger/index.html), I propose to add the parameter `respect.unordered.factors`:

Unordered categorical predictors should be handled in 3 different ways by using `respect.unordered.factors`: For 'ignore' all factors are regarded ordered, for 'partition' all possible 2-partitions are considered for splitting. For 'order' and 2-class classification the factor levels are ordered by their proportion falling in the second class, for regression by their mean response, as described in Hastie et al. (2009), chapter 9.2.4. The use of 'order' is recommended, as it computationally fast and can handle an unlimited number of factor levels. Note that the factors are only reordered once and not again in each split.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

The combinatorial search can be avoided in the case of binary classification or regression as shown by Breiman in his original work.
user: markusloecher
issue_21921: state: open
title: DummyClassifier does not pass check_estimator
body: ### Describe the bug

Hi, the `sklearn.dummy.DummyClassifier` fails when tested with `sklearn.utils.estimator_checks.check_estimator`.
I expect it to pass, for example with parameter `strategy="prior"`, or `strategy="most_frequent"`.

I know my scikit-learn version is not up-to-date (`sklearn version 0.24.2`). But I still expected it to work on any version, or at least for relatively recent versions (such as mine, `0.24.2`).

Thanks for any correction and help!

### Steps/Code to Reproduce

```python
from sklearn.dummy import DummyClassifier
from sklearn.utils.estimator_checks import check_estimator

check_estimator(DummyClassifier(strategy="most_frequent"))
```

### Expected Results

No error thrown. Test is passed.

### Actual Results

```python-traceback
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-19-2a6cfb0297f4> in <module>
      2 from sklearn.utils.estimator_checks import check_estimator
      3 
----> 4 check_estimator(DummyClassifier(strategy="most_frequent"))

~/code/scikit-learn/sklearn/utils/estimator_checks.py in check_estimator(Estimator, generate_only)
    596     for estimator, check in checks_generator():
    597         try:
--> 598             check(estimator)
    599         except SkipTest as exception:
    600             # SkipTest is thrown when pandas can't be imported, or by checks

~/code/scikit-learn/sklearn/utils/estimator_checks.py in check_classifiers_classes(name, classifier_orig)
   2717         for y_names_i in [y_names, y_names.astype("O")]:
   2718             y_ = _choose_check_classifiers_labels(name, y, y_names_i)
-> 2719             check_classifiers_predictions(X, y_, name, classifier_orig)
   2720 
   2721     labels_binary = [-1, 1]

~/code/scikit-learn/sklearn/utils/_testing.py in wrapper(*args, **kwargs)
    311             with warnings.catch_warnings():
    312                 warnings.simplefilter("ignore", self.category)
--> 313                 return fn(*args, **kwargs)
    314 
    315         return wrapper

~/code/scikit-learn/sklearn/utils/estimator_checks.py in check_classifiers_predictions(X, y, name, classifier_orig)
   2665         # For some specific cases 'ComplementNB' predicts less classes
   2666         # than expected
-> 2667         assert_array_equal(np.unique(y), np.unique(y_pred))
   2668     assert_array_equal(
   2669         classes,

    [... skipping hidden 1 frame]

~/mambaforge/envs/dev/lib/python3.9/site-packages/numpy/testing/_private/utils.py in assert_array_compare(comparison, x, y, err_msg, verbose, header, precision, equal_nan, equal_inf)
    761                                 verbose=verbose, header=header,
    762                                 names=('x', 'y'), precision=precision)
--> 763             raise AssertionError(msg)
    764 
    765         flagged = bool_(False)

AssertionError: 
Arrays are not equal

(shapes (2,), (1,) mismatch)
 x: array(['one', 'two'], dtype='<U3')
 y: array(['one'], dtype='<U3')
```

### Versions

`sklearn version 0.24.2`
user: BrunoBarre
issue_21895: state: open
title: GridSearchCV & RandomizedSearchCV has a ranking issue when test scores are same.
body: ### Describe the bug

During best parameter candidate search when two parameter candidates have the same `mean_test_<scorer_name>` then the `rank_test_<scorer_name>` ranks them the same, and if both have rank 1 then the algorithm picks the one that come first.

### Steps/Code to Reproduce

```python
from sklearn import svm, datasets
import pandas as pd
import numpy as np
from sklearn.model_selection import GridSearchCV
iris = datasets.load_iris()
def grid_search(parameters):
    svc = svm.SVC()
    clf = GridSearchCV(svc, parameters,return_train_score=True)
    clf.fit(iris.data, iris.target)
    print(pd.DataFrame(clf.cv_results_)[['param_C','param_kernel','mean_test_score','rank_test_score','mean_train_score']])
    print('best_parameters = ',clf.best_params_,'\n')

params = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
grid_search(params)
params = {'kernel':('rbf', 'linear'), 'C':[10, 1]}
grid_search(params)
```

### Expected Results

Both function calls should return the same output for `clf.best_params_` 
i.e  `best_parameters =  {'C': 1, 'kernel': 'linear'}`  because the difference between the `mean_train_score` and the `mean_test_score` is the least for this parameter candidate.

### Actual Results

![scikit-learn_bug](https://user-images.githubusercontent.com/20677278/144828758-b6b8820a-d595-4194-8cb1-246606b91925.JPG)

### Versions

```

System:
    python: 3.9.7 (default, Sep 16 2021, 16:59:28) [MSC v.1916 64 bit (AMD64)]
executable: C:\Anaconda3\anaconda3\envs\sklearndev\python.exe
   machine: Windows-10-10.0.19043-SP0

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 1.1.dev0
        numpy: 1.21.2
        scipy: 1.7.1
       Cython: 0.29.24
       pandas: 1.3.4
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 3.0.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: mkl
         prefix: mkl_rt
       filepath: C:\Anaconda3\anaconda3\envs\sklearndev\Library\bin\mkl_rt.1.dll
        version: 2021.4-Product
threading_layer: intel
    num_threads: 2

user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: C:\Windows\System32\vcomp140.dll
        version: None
    num_threads: 4
```
user: AChand20
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_21891: state: open
title: Lasso(alpha) = SGDRegressor(alpha, penalty="l1"), but Ridge(alpha) = SGDRegressor(alpha / len(train_set))
body: ### Describe the bug

`Ridge` and `SGDRegressor` are inconsistent in the way they handle the regularization hyperparameter `alpha`:

* `Ridge(alpha=alpha)` is equivalent to `SGDRegressor(alpha / m, penalty="l2", ...)`, where `m` is the training set size.
* But `Lasso(alpha=alpha)` is equivalent to `SGDRegressor(alpha, penalty="l1", ...)`, which is great.

Looking at the source code of `SGDRegressor`, it looks like `l1` and `l2` are consistent, so it must be a problem with `Ridge`. Perhaps it should be modified to divide `alpha` by `m`?

Is it just historical, or is there a rationale behind this?


### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.linear_model import Lasso, Ridge, SGDRegressor

alpha = 0.1
m = 100

np.random.seed(42)
X = 2 * np.random.rand(m, 1)
y = 4 + 3 * X + np.random.randn(m, 1)

ridge = Ridge(alpha=alpha)
ridge.fit(X, y)
print("Ridge:", ridge.intercept_, ridge.coef_)

sgd_reg = SGDRegressor(
                       loss="squared_error", penalty="l2", alpha=alpha / m,
                       max_iter=10_000, tol=None, eta0=0.0001,
                       learning_rate="constant", random_state=42)
sgd_reg.fit(X, y.ravel())
print("SGDRegressor with l2:", sgd_reg.intercept_, sgd_reg.coef_)

lasso = Lasso(alpha=alpha)
lasso.fit(X, y)
print("Lasso:", lasso.intercept_, lasso.coef_)

sgd_reg = SGDRegressor(loss="squared_error", penalty="l1", alpha=alpha,
                       max_iter=10_000, tol=None, eta0=0.0001,
                       learning_rate="constant", random_state=42)
sgd_reg.fit(X, y.ravel())
print("SGDRegressor with l1:", sgd_reg.intercept_, sgd_reg.coef_)
```


### Expected Results

I would expect the `Ridge` model to be different from the `SGDRegressor` model, since it does not use the same `alpha` value. Instead, they are equal.

### Actual Results

But here is the output of the previous code:

```
Ridge: [4.22250784] [[2.76223165]]
SGDRegressor with l2: [4.22261191] [2.76235098]
Lasso: [4.48341837] [2.48477396]
SGDRegressor with l1: [4.4834874] [2.48492104]
```

As you can see, the `Ridge` model is equivalent to `SGDRegressor(alpha=alpha / m, ...)` instead of `SGDRegressor(alpha=alpha, ...)`. But `Lasso(alpha)` is equivalent to `SGDRegressor(alpha=alpha, penalty="l1", ...)`.

### Versions

Running on Colab:

```
System:
    python: 3.7.12 (default, Sep 10 2021, 00:21:48)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.1
        numpy: 1.19.5
        scipy: 1.4.1
       Cython: 0.29.24
       pandas: 1.1.5
   matplotlib: 3.2.2
       joblib: 1.1.0
threadpoolctl: 3.0.0

Built with OpenMP: True
```
user: ageron
labels: Needs Decision - API: Requires decision regarding API
Needs Decision - Backward Compatibility: Requires decision regarding backward compatibility
issue_21876: state: open
title: LocalOutlierFactor exposes methods that it shouldn't
body: ### Describe the issue linked to the documentation

The LocalOutlierFactor class (and hence documentation) exposes methods that it probably shouldn't:

https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.LocalOutlierFactor.html#sklearn.neighbors.LocalOutlierFactor.kneighbors
https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.LocalOutlierFactor.html#sklearn.neighbors.LocalOutlierFactor.kneighbors_graph

these appear to be inherited from the parent class, and should not be exposed.

### Suggest a potential alternative/fix

Rather than using `KNeighborsMixin` and `NeighborsBase`, the implementation should probably just use an instance of a nearest neighbor search internally, and not expose this as part of the local outlier factor API.
user: kno10
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_21875: state: open
title: Non-standard "negative_outlier_factor_"
body: ### Describe the bug

The scikit-learn implementation uses `negative_outlier_factor_`, which seems to be simply the negative of the standard local outlier factor. As I understand the objectives of scikit-learn, I thought the project usually prefers to implement the standard version, I have not yet seen any literature that uses such negative factors.
What is the motivation for this? It tends to confuse users (c.f., #9874).

### Steps/Code to Reproduce

-

### Expected Results

-

### Actual Results

-

### Versions

-
user: kno10
issue_21861: state: open
title: MultiOutputClassifier should have `decision_function`
body: ### Describe the workflow you want to enable

We should be able to use `MultiOutputClassifier` as (essentially) a drop-in replacement for the underlying classifiers. Right now it has the functions `predict`  and `predict_proba` (if defined in the underlying predictors) but missing `decision_function` (e.g. for svm). 

### Describe your proposed solution

Use exactly the same code as `predict_proba` :
https://github.com/scikit-learn/scikit-learn/blob/ec48b246785f37f3d976693c03dc5a3702ccedde/sklearn/multioutput.py#L450
but for `decision_function`


user: YannDubs
issue_21846: state: open
title: RandomizedSearchCV does not properly parse grid parameters when MultiInputOutputRegressor wrapper is used over a scikit learn pipeline.
body: ### Describe the bug

The documentation of scikit for the multiple input output wrappers is actually not that good but it is mentioned that:

https://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputRegressor.html

```
set_params(**params)[source]¶
Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). 
The latter have parameters of the form <component>__<parameter> so that it’s possible to
update each component of a nested object.
```
Therefore I am constructing my scikit learn pipeline as usual and I am setting the hyperparameters with a dictionary that it is constructed by name_of_my_estimator_step_in_the_pipeline__param1 etc...

When running the RandomizedSearchCV I can not get results.

```
parameters = dict({
    'estimator__max_depth': [10, 100, 1000]
})

mimo_wrapper.estimator.set_params(estimator__max_depth=200)

Pipeline(steps=[('scaler', StandardScaler()),
                ('variance_selector', VarianceThreshold(threshold=0.03)),
                ('estimator',
                 XGBRegressor(base_score=None, booster=None,
                              colsample_bylevel=None, colsample_bynode=None,
                              colsample_bytree=None, gamma=None, gpu_id=None,
                              importance_type='gain',
                              interaction_constraints=None, learning_rate=None,
                              max_delta_step=None, max_depth=200,
                              min_child_weight=None, missing=nan,
                              monotone_constraints=None, n_estimators=100,
                              n_jobs=None, num_parallel_tree=None,
                              random_state=None, reg_alpha=None,
                              reg_lambda=None, scale_pos_weight=None,
                              subsample=None, tree_method=None,
                              validate_parameters=None, verbosity=None))])
```

I am doing something wrong or is this a bug? 
BR
/Edgar



### Steps/Code to Reproduce

**Case 1:**

```python
pipeline_xgboost = Pipeline([('scaler', StandardScaler()),
                             ('variance_selector', VarianceThreshold(threshold=0.03)), 
                             ('estimator', xgb.XGBRegressor())])
```
And then creating the wrapper as:

```python 
mimo_wrapper = MultiOutputRegressor(pipeline_xgboost)
```

Following the documentation of scikit pipelines **I was** defining my xgboost parameters as:

```python
parameters = [
    {
        'estimator__reg_alpha': [0.0001, 0.001, 0.01, 0.1, 1, 10, 100],
        'estimator__max_depth': [10, 100, 1000]
         etc...
    }

randomized_search = RandomizedSearchCV(mimo_wrapper, perparameters, random_state=0, n_iter=5,
                                       n_jobs=-1, refit=True, cv=3, verbose=True,
                                       pre_dispatch='2*n_jobs', error_score='raise', 
                                       return_train_score=True,
                                       scoring='neg_mean_absolute_error')

hyperparameters_tuning = random_grid.fit(df.drop(columns=TARGETS+UMAPS),
                              df[TARGETS])
```

**Case 2:**


```python

parameters = {
    'estimator__estimator__reg_alpha': [0.0001, 0.001, 0.01, 0.1, 1, 10, 100],
    'estimator__estimator__max_depth': [10, 100, 1000]
}


random_grid = RandomizedSearchCV(estimator=pipeline_xgboost, param_distributions=parameters,random_state=0, n_iter=5,
                                       n_jobs=-1, refit=True, cv=3, verbose=True,
                                       pre_dispatch='2*n_jobs', error_score='raise', 
                                       return_train_score=True,
                                       scoring='neg_mean_absolute_error')

hyperparameters_tuning = random_grid.fit(df.drop(columns=TARGETS+UMAPS),
                              df[TARGETS])
```

### Expected Results

No error is thrown

### Actual Results

Case 1:

```python
ValueError: Invalid parameter reg_alpha for estimator Pipeline(steps=[('scaler', StandardScaler()),
                ('variance_selector', VarianceThreshold(threshold=0.03)),
                ('estimator',
                 XGBRegressor(base_score=None, booster=None,
                              colsample_bylevel=None, colsample_bynode=None,
                              colsample_bytree=None, gamma=None, gpu_id=None,
                              importance_type='gain',
                              interaction_constraints=None, learning_rate=None,
                              max_delta_step=None, max_depth=None,
                              min_child_weight=None, missing=nan,
                              monotone_constraints=None, n_estimators=100,
                              n_jobs=None, num_parallel_tree=None,
                              random_state=None, reg_alpha=None,
                              reg_lambda=None, scale_pos_weight=None,
                              subsample=None, tree_method=None,
                              validate_parameters=None, verbosity=None))]). Check the list of available parameters with `estimator.get_params().keys()`.
```



Case 2: 
```python

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
/tmp/ipykernel_11898/2539017483.py in <module>
----> 1 hyperparameters_tuning = random_grid.fit(final_file_df_with_aggregates.drop(columns=TARGETS+UMAPS),
      2                               final_file_df_with_aggregates[TARGETS])

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/sklearn/model_selection/_search.py in fit(self, X, y, groups, **fit_params)
    889                 return results
    890 
--> 891             self._run_search(evaluate_candidates)
    892 
    893             # multimetric is determined here because in the case of a callable

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/sklearn/model_selection/_search.py in _run_search(self, evaluate_candidates)
   1764     def _run_search(self, evaluate_candidates):
   1765         """Search n_iter candidates from param_distributions"""
-> 1766         evaluate_candidates(
   1767             ParameterSampler(
   1768                 self.param_distributions, self.n_iter, random_state=self.random_state

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/sklearn/model_selection/_search.py in evaluate_candidates(candidate_params, cv, more_results)
    836                     )
    837 
--> 838                 out = parallel(
    839                     delayed(_fit_and_score)(
    840                         clone(base_estimator),

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/joblib/parallel.py in __call__(self, iterable)
   1054 
   1055             with self._backend.retrieval_context():
-> 1056                 self.retrieve()
   1057             # Make sure that we get a last message telling us we are done
   1058             elapsed_time = time.time() - self._start_time

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/joblib/parallel.py in retrieve(self)
    933             try:
    934                 if getattr(self._backend, 'supports_timeout', False):
--> 935                     self._output.extend(job.get(timeout=self.timeout))
    936                 else:
    937                     self._output.extend(job.get())

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/joblib/_parallel_backends.py in wrap_future_result(future, timeout)
    540         AsyncResults.get from multiprocessing."""
    541         try:
--> 542             return future.result(timeout=timeout)
    543         except CfTimeoutError as e:
    544             raise TimeoutError from e

/anaconda/envs/azureml_py38/lib/python3.8/concurrent/futures/_base.py in result(self, timeout)
    437                 raise CancelledError()
    438             elif self._state == FINISHED:
--> 439                 return self.__get_result()
    440             else:
    441                 raise TimeoutError()

/anaconda/envs/azureml_py38/lib/python3.8/concurrent/futures/_base.py in __get_result(self)
    386     def __get_result(self):
    387         if self._exception:
--> 388             raise self._exception
    389         else:
    390             return self._result
```


### Versions

```
System:
    python: 3.8.1 (default, Jan  8 2020, 22:29:32)  [GCC 7.3.0]
executable: /anaconda/envs/azureml_py38/bin/python
   machine: Linux-5.4.0-1056-azure-x86_64-with-glibc2.10

Python dependencies:
          pip: 20.1.1
   setuptools: 50.3.0.post20201006
      sklearn: 1.0.1
        numpy: 1.21.2
        scipy: 1.6.2
       Cython: 0.29.21
       pandas: 1.3.0
   matplotlib: 3.4.3
       joblib: 1.1.0
threadpoolctl: 2.1.0

Built with OpenMP: True
```
user: edgBR
issue_21841: state: open
title: TST make sure to have a real multioutput target in common tests
body: Our common test creates an array of shape `(n_samples, 1)` for the multioutput problem.
However, this problem would be considered as a single output problem if `type_of_target` is used in one of the estimators.

This test introduce a real multi output problem with `y` of shape `(n_samples, 2)`
user: glemaitre
issue_21840: state: open
title: Intercept in Kernel Ridge Regression
body: ### Describe the workflow you want to enable

Currently, Kernel Ridge Regression as implemented in `sklearn.kernel_ridge.KernelRidge` does not support an intercept like e.g. `sklearn.svm.SVR` does. This can lead to problems if the target value is shifted.

### Describe your proposed solution

Implement an intercept into `KernelRidge`.

### Describe alternatives you've considered, if relevant

Document that there is no intercept present for `KernelRidge` and that, therefore, manual target value shifting is necessary. This is of course the easier path with less work and maintenance.

### Additional context

[Demo of problems caused by missing intercept](https://gist.github.com/Turakar/e451b0236e9729cfde759aa7eb29d8ea)
user: Turakar
issue_21839: state: open
title: FIX make sure to limit ClassifierChain to multilabel-indicator target
body: Requires to solve #21841 before.

`ClassifierChain` should be limited to `multilabel-indicator`. Fitting a `multiclass-mulltiouput` and call `.decision_function` will lead to an assignment error.

In addition, the common tests are not modifying the `y` properly when considering `multioutput`/`multilabel` estimator. One should indeed make sure to have 2 targets (a single target in a 2d array, would be considered as `binary` or `multiclass` by our `type_of_target`).

This is necessary to be able to run `check_estimators_dtypes` on `ClassifierChain` (and estimators in general). Currently the test is not ran.
user: glemaitre
issue_21818: state: open
title: MinCovDet can't deal with singular matrix
body: ### Describe the bug

`MinCovDet` can't deal with singular covariance, for example in the zero covariance case. Note that usual `EmpiricalCovariance` can deal with that case too.

### Steps/Code to Reproduce

```python
from sklearn.covariance import EmpiricalCovariance, MinCovDet
import numpy as np

data = np.zeros((3, 2))
print(EmpiricalCovariance().fit(data).covariance_)
print(MinCovDet().fit(data).covariance_)
```

### Expected Results

[[0. 0.]
 [0. 0.]]
[[0. 0.]
 [0. 0.]]

### Actual Results

```
[[0. 0.]
 [0. 0.]]
/home/carlos/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/lib/python3.7/site-packages/sklearn/covariance/_robust_covariance.py:739: UserWarning: The covariance matrix associated to your dataset is not full rank
  "The covariance matrix associated to your dataset is not full rank"
/home/carlos/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/lib/python3.7/site-packages/sklearn/covariance/_robust_covariance.py:806: RuntimeWarning: invalid value encountered in true_divide
  self.dist_ /= correction
/home/carlos/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/lib/python3.7/site-packages/sklearn/covariance/_robust_covariance.py:848: RuntimeWarning: Mean of empty slice.
  location_reweighted = data[mask].mean(0)
/home/carlos/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/lib/python3.7/site-packages/numpy/core/_methods.py:182: RuntimeWarning: invalid value encountered in true_divide
  ret, rcount, out=ret, casting='unsafe', subok=False)
/home/carlos/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/lib/python3.7/site-packages/numpy/lib/function_base.py:380: RuntimeWarning: Mean of empty slice.
  avg = a.mean(axis)
/home/carlos/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/lib/python3.7/site-packages/sklearn/covariance/_empirical_covariance.py:94: RuntimeWarning: Degrees of freedom <= 0 for slice
  covariance = np.cov(X.T, bias=1)
/home/carlos/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/lib/python3.7/site-packages/numpy/lib/function_base.py:2542: RuntimeWarning: divide by zero encountered in true_divide
  c *= np.true_divide(1, fact)
/home/carlos/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/lib/python3.7/site-packages/numpy/lib/function_base.py:2542: RuntimeWarning: invalid value encountered in multiply
  c *= np.true_divide(1, fact)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-34-037a31d6709d> in <module>
      3 data = np.zeros((3, 2))
      4 print(EmpiricalCovariance().fit(data).covariance_)
----> 5 print(MinCovDet().fit(data).covariance_)

~/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/lib/python3.7/site-packages/sklearn/covariance/_robust_covariance.py in fit(self, X, y)
    763         self.correct_covariance(X)
    764         # re-weight estimator
--> 765         self.reweight_covariance(X)
    766 
    767         return self

~/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/lib/python3.7/site-packages/sklearn/covariance/_robust_covariance.py in reweight_covariance(self, data)
    852         support_reweighted = np.zeros(n_samples, dtype=bool)
    853         support_reweighted[mask] = True
--> 854         self._set_covariance(covariance_reweighted)
    855         self.location_ = location_reweighted
    856         self.support_ = support_reweighted

~/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/lib/python3.7/site-packages/sklearn/covariance/_empirical_covariance.py in _set_covariance(self, covariance)
    185             is computed.
    186         """
--> 187         covariance = check_array(covariance)
    188         # set covariance
    189         self.covariance_ = covariance

~/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/lib/python3.7/site-packages/sklearn/utils/validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
    790 
    791         if force_all_finite:
--> 792             _assert_all_finite(array, allow_nan=force_all_finite == "allow-nan")
    793 
    794     if ensure_min_samples > 0:

~/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/lib/python3.7/site-packages/sklearn/utils/validation.py in _assert_all_finite(X, allow_nan, msg_dtype)
    114             raise ValueError(
    115                 msg_err.format(
--> 116                     type_err, msg_dtype if msg_dtype is not None else X.dtype
    117                 )
    118             )

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
```

### Versions

System:
    python: 3.7.6 (default, Jan  8 2020, 19:59:22)  [GCC 7.3.0]
executable: /home/carlos/Programas/Utilidades/Lenguajes/miniconda3/envs/fda/bin/python
   machine: Linux-5.4.0-90-generic-x86_64-with-debian-buster-sid

Python dependencies:
          pip: 21.3.1
   setuptools: 58.5.2
      sklearn: 1.0.1
        numpy: 1.21.2
        scipy: 1.6.2
       Cython: 0.29.14
       pandas: 1.3.4
   matplotlib: 3.4.2
       joblib: 1.1.0
threadpoolctl: 3.0.0

Built with OpenMP: True

user: vnmabus
issue_21815: state: open
title: Sparse NeighborhoodComponentsAnalysis
body: ### Describe the workflow you want to enable

Text classification using KNN on text features concatenated with other features. As the dimentionality of text features can be very high, a good way to reduce the dimensionality is using NCA.

Example pipeline:
```python
from sklearn.neighbors import NeighborhoodComponentsAnalysis as NCA, KNeighborsClassifier as KNNC
...
text_pipeline = Pipeline(steps=[
        ('tfidf', TfidfVectorizer()),  
        ('todense', FunctionTransformer(lambda x: x.todense(), accept_sparse=True)),   # <--- Feature Request !
        ('nca', NCA(n_components=text_components)),
    ])

# additional_ft is a list of columns from the data that are appended to the text features.
transformers = [('text_pipeline', text_pipeline, 'Text Column'), ('other_ft', 'passthrough', additional_ft)]
feature_pipelines = ColumnTransformer(transformers)
model = Pipeline(steps=[ColumnTransformer(transformers), ('knnc', KNNC(n_neighbors=3, n_jobs=-1))])
```

Today, NCA does not support Sparse Matrices, so the tfidf features have to be set to dense first. This is a costly operation (in memory). 

### Describe your proposed solution

Add support for sparse matrices, similar to `SparsePCA`.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: thomassajot
issue_21811: state: open
title: FIX Sets feature_names_in_ for estimators in Bagging*
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #21599

#### What does this implement/fix? Explain your changes.
This PR sets `feature_names_in_` for all inner estimators in `Bagging*`. I placed this in 1.0.2 for now, but open to move it to 1.1.

#### Any other comments?
Should we set `feature_names_in_` for all meta-estimators that needs to validate first?

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
issue_21804: state: open
title: RFC / API add option to fit/predict without input validation
body: ### Goal
Ideally, this issue helps to sort out an API.

### Describe the workflow you want to enable

I'd like to be able to switch on/off:

1. Parameter validation in `fit` 
2. Input array validation in `fit` and `predict`
3. All other validation steps in `fit` and `predict` (e.g. `check_is_fitted`)

Something like
```python
model = RandomForestRegressor(validate_params=False)
model.fit(validate_input=False)
model.predict(validated_input=False)
```

Note that some estimators like `Lasso` already support `check_input` in `fit`.

The main reason to do so is improved performance.

### Additional context

Related issues and PRs are #16653, #20657, #21578 (in particular https://github.com/scikit-learn/scikit-learn/pull/21578#discussion_r745491121)
Related discussions: https://github.com/scikit-learn/scikit-learn/discussions/21810
user: lorentzenchr
issue_21801: state: open
title: [WIP] CountVectorizer and TfidfVectorizer option to featurize out-of-vocab.
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

Tasks:
- [x] Implement out-of-vocab count feature (`out_of_vocab_features=1`)
- [x] Implement out-of-vocab hashbag count feature (`out_of_vocab_features>1`)
- [ ] Add documentation and example
- [x] Add option version number
- [x] Added `out_of_vocab_features` option to `TfidfVectorizer` as well
- [x] Unit tests
- [x] Implement `inverse_transform` for added features
- [ ] Example cases where this feature improves outcome versus without these features

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes [#21790](https://github.com/scikit-learn/scikit-learn/issues/21790).


#### What does this implement/fix? Explain your changes.
Adds `out_of_vocab_features` option to `CountVectorizer`:
```
    out_of_vocab_features : int, default=None
        If not None, will add the specified number of out-of-vocabulary
        features counting the terms not in the vocabulary.
        If set to 1, it will add a single feature as total count of the
        out-of-vocab terms found per document.
        If set to >1, on top of the single count feature above it will also
        add the specified number of features to represent all out-of-vocab
        terms through overlapping feature hashing count buckets. For example,
        a value of 1024 will add single feature counting the total out-of-
        vocabulary terms, plus 1024 features from a hash-bag counting
        out-of-vocabulary terms in overlapping buckets. Note hash bag uses
        Murmurhash3 and the out-of-vocab terms will overlap into these
        count buckets.
        This option is often used in conjunction with feature trimming
        `max_features`, `min_df`, `max_df`, and/or `vocabulary` options to
        featurize the trimmed-off terms in a compact form.
```

Many tabular data models run CountVectorizer against categorical and text data to featurize the data. At the large scale especially, the min_df or max_features are commonly used to limit the dimensionality to the most frequent terms. This is important to reduce memory and sometimes may help with generalization of the resulting model.

One problem is that sometimes this long-tail of low-count terms has useful information, and researchers sometimes instead replace these features with a flag indicating that the feature was filtered out. This can further be extended to not only replace it with a single-feature denoting out of dictionary, but a hashbag of N terms denoting the long tail. Eg. a hashbag of 100 terms could be used to denote the long-tail of out of dictionary terms.

Example tests of current version:
```python
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
corpus = [
    'This is the first document test test.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?',
]


# Test usage without out-of-vocabulary features
vectorizer = CountVectorizer(analyzer='word', ngram_range=(1, 1), min_df=2)
X = vectorizer.fit_transform(corpus)

print(vectorizer.get_feature_names_out())
print(X.toarray())

'''
Output:
['document' 'first' 'is' 'the' 'this']
[[1 1 1 1 1]
 [2 0 1 1 1]
 [0 0 1 1 1]
 [1 1 1 1 1]]
'''

# Test usage with a single out-of-vocabulary feature count
vectorizer = CountVectorizer(analyzer='word', ngram_range=(1, 1), min_df=2, out_of_vocab_features=1)
X = vectorizer.fit_transform(corpus)

print(vectorizer.get_feature_names_out())
print(X.toarray())

'''
Output:
['document' 'first' 'is' 'the' 'this']
[[1 1 1 1 1 2]
 [2 0 1 1 1 1]
 [0 0 1 1 1 3]
 [1 1 1 1 1 0]]
'''

# Test usage with a eight out-of-vocabulary feature counts
vectorizer = CountVectorizer(analyzer='word', ngram_range=(1, 1), min_df=2, out_of_vocab_features=8)
X = vectorizer.fit_transform(corpus)

print(vectorizer.get_feature_names_out())
print(X.toarray())

'''
Output:
['document' 'first' 'is' 'the' 'this']
[[1 1 1 1 1 2 0 0 0 0 0 2 0 0]
 [2 0 1 1 1 1 0 0 0 1 0 0 0 0]
 [0 0 1 1 1 3 0 0 0 0 1 2 0 0]
 [1 1 1 1 1 0 0 0 0 0 0 0 0 0]]
'''
```

#### Any other comments?

I added `out_of_vocab_features` to the end of the parameters of `CountVectorizer` to be fully backwards compatible. Style-wise it makes more sense after `max_features` and before `vocabulary` in my opinion. Thoughts if I should I leave this at the end for compatibility, or place the new option more logically?

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: glmcdona
issue_21790: state: open
title: CountVectorizer out of dictionary features
body: ### Describe the workflow you want to enable

Many tabular data models run CountVectorizer against categorical and text data to featurize the data. At the large scale especially, the `min_df` or `max_features` are commonly used to limit the dimensionality to the most frequent terms. This is important to reduce memory and sometimes may help with generalization of the resulting model.

One problem is that sometimes this long-tail of low-count terms has useful information, and researchers sometimes instead replace these features with a flag indicating that the feature was filtered out. This can further be extended to not only replace it with a single-feature denoting out of dictionary, but a hashbag of N terms denoting the long tail. Eg. a hashbag of 100 terms could be used to denote the long-tail of out of dictionary terms.

### Describe your proposed solution

Update CountVectorizer to enable out-of-dictionary features. Proposed modification stub might be:

```
CountVectorizer(..., out_of_dict_features=1)
```

And docs:
```
    out_of_dict_features : int, default=None
        If not None, will add the specified number of out-of-dictionary
        features. If it is 1, it will add a single feature to denote
        out of dictionary features. If >1, it will add a hashbag of the
        specified size to denote the out of dictionary features. This
        should be used with feature trimming `max_features`, `min_df`,
        and/or `max_df`.
```

Note: I may change this to be the number of bits to use for the out of dict hashbag instead of the feature count.

Example with single out of dictionary feature:
```
from sklearn.feature_extraction.text import CountVectorizer
corpus = [
    'This is the first document test.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?',
]

vectorizer = CountVectorizer(analyzer='word', ngram_range=(1, 1), min_df=2, out_of_dict_features=1)
X = vectorizer.fit_transform(corpus)

print(vectorizer.get_feature_names_out())
print(X.toarray())
```

Mock output:
```
['document' 'first' 'is' 'the' 'this' '<out_of_dict_0>']
[[1 1 1 1 1 1]
 [2 0 1 1 1 1]
 [0 0 1 1 1 3]
 [1 1 1 1 1 0]]
```
The last column in the above is the count of out-of-dictionary features.

Example with multiple out-of-dict features:
```
from sklearn.feature_extraction.text import CountVectorizer
corpus = [
    'This is the first document test.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?',
]

vectorizer = CountVectorizer(analyzer='word', ngram_range=(1, 1), min_df=2, out_of_dict_features=100)
X = vectorizer.fit_transform(corpus)

print(vectorizer.get_feature_names_out())
print(X.toarray())
```

Mock output:
```
['document' 'first' 'is' 'the' 'this' '<out_of_dict_79>']
[[1 1 1 1 1 <100x bag count of out of dictionary features appended>]
 [2 0 1 1 1 <100x bag count of out of dictionary features appended>]
 [0 0 1 1 1 <100x bag count of out of dictionary features appended>]
 [1 1 1 1 1 <100x bag count of out of dictionary features appended>]]
```

### Describe alternatives you've considered, if relevant

This is difficult to implement in production without this change where you don't know some of the never-seen out of dictionary terms that it may be inferenced against.

### Additional context

I'm starting to code a PR to add this feature. Let me know if there is any feedback, requests, or comments!
user: glmcdona
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_21786: state: open
title: sklearn.cluster.OPTICS uses all cores when minkowski's p=2
body: ### Describe the bug

I have specified n_jobs=2. When p=1, joblib backend is working as specified. But when p=2 it ignores n_jobs and goes full power.

### Steps/Code to Reproduce

optics_oh = OPTICS(min_samples=3, p=2, n_jobs=2)
optics_oh_predict = optics_oh.fit_predict(X_oh)

### Expected Results

Two cores loaded

### Actual Results

All cores loaded
![Screenshot from 2021-11-25 15-23-14](https://user-images.githubusercontent.com/78765372/143458868-f827f19a-8657-4d61-90ac-933826aca627.png)


### Versions

System:
    python: 3.8.10 (default, Sep 28 2021, 16:10:42)  [GCC 9.3.0]
executable: /home/some_folder_structure/env/bin/python
   machine: Linux-5.11.0-40-generic-x86_64-with-glibc2.29

Python dependencies:
          pip: 21.3.1
   setuptools: 58.2.0
      sklearn: 0.24.2
        numpy: 1.21.4
        scipy: 1.5.4
       Cython: 0.29.22
       pandas: 1.2.2
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
user: szperajacyzolw
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_21784: state: open
title: Initial implementation of GridFactory
body: 
#### Reference Issues/PRs
Fixes #19045

#### What does this implement/fix? Explain your changes.

This implements a GridFactory with functionality comparable to [searchgrid](https://github.com/jnothman/searchgrid) to allow easier specification of parameter search spaces, especially as a composite estimator is built.

#### Any other comments?


A complete version of this will:
* [ ] import tests from searchgrid
* [ ] adopt GridFactory in some examples
* [ ] add a DistributionFactory for use with Randomized searches (this is tricky because currently Randomized searches do not support conditional search spaces, unless I'm much mistaken)
* [ ] add factory support to *SearchCV (currently only GridSearchCV)
* [ ] maybe add a function to interpret the parameter names in `cv_results_` in a `__`-free way

I note that supporting a factory to be passed to `GridSearchCV` instead of a parameter grid doesn't save the user much code. Maybe we don't need to support it?

user: jnothman
issue_21783: state: open
title: Enhance semi_supervised.LabelPropagation
body: ### Describe the workflow you want to enable

In the seminal paper quoted as a reference for the implementation of `semi_supervised.label_propagation` : [Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. Technical Report CMU-CALD-02-107, Carnegie Mellon University, 2002](http://pages.cs.wisc.edu/~jerryzhu/pub/CMU-CALD-02-107.pdf), **three tricks** are used to increase the performance that are not currently implemented in the code. 

## Row normalization
Page 4 : it seems that instead of row-normalizing the Y matrix at each iteration, we can simply row-normalize the transition matrix at the beginning. This may save some time (to be benchmarked) 
![image](https://user-images.githubusercontent.com/34744445/143325613-3a3e7c3a-38d5-4a1a-b769-644b04232828.png)
![image](https://user-images.githubusercontent.com/34744445/143325654-b473644d-b94a-45f5-bb49-ee76e09b50c1.png)

## Post-processing
Page 6: authors use a trick to ensure that the distribution of classes in the label propagated prediction is similar to the one in the labeled data. To do so, they use either Class Mass Normalization (CMN) or Label Bidding (LB). According to their experiments (page 17), CMN provides a significant improvement over non-post-processed version, whereas LB is slightly less good. Also, it depends on whether the true class distribution is known in advance or estimated from the available labels. 
![image](https://user-images.githubusercontent.com/34744445/143326238-16343b33-1ae9-470d-a562-ef67b6937e6c.png)

## Smoothing the transition matrix.
Page 10 : it seems that adding a smoothing parameter to add some uniform transition probability across the transition matrix is beneficial, although the paper doesn't show any benchmark proving the superiority of this approach. 
![image](https://user-images.githubusercontent.com/34744445/143326040-fb6df5ba-83fb-40e8-b052-3a0e4cbcced7.png)



### Describe your proposed solution

We may want to benchmark the proposed features to see if they improve the performance on real datasets, and implement them if applicable.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: pedugnat
issue_21774: state: open
title: Calibration and Refinement loss for Brier score loss
body: ### Describe the workflow you want to enable

As per [Brier score User Guide](https://scikit-learn.org/stable/modules/model_evaluation.html#brier-score-loss):

“Only when refinement loss remains the same does a lower Brier score loss always mean better calibration”

But the current API doesn't provide refinement loss/calibration loss. 
Which makes it hard to measure the quality of probabilistic estimates.

### Describe your proposed solution

My proposed solution involves implementing the one described in the paper [[Flach2008]
](https://drops.dagstuhl.de/opus/volltexte/2008/1382/pdf/07161.FlachPeter.Paper.1382.pdf), the reference which is conveniently already mentioned in the [User Guide](https://scikit-learn.org/stable/modules/model_evaluation.html#flach2008)

Namely, estimating Calibration loss and Refinement loss without any binning, on raw data/predictions. 

If the community decides this is a valuable addition, I humbly present [my implementation](https://github.com/ColdTeapot273K/scikit-learn/blob/brier-score-binless-decomposition/sklearn/metrics/_classification.py#L2693-L2790), which is WIP in terms of Scikit-Learn codebase conventions compliance and corner case processing, but is functioning in essence.

### Describe alternatives you've considered, if relevant

Visual comparison of calibration curves 
<not exact, not scalable

Making custom ad hoc metrics to estimate probability errors
<fragile, dubious

Using `sklearn.calibration.calibration_curve`
<requires binning<sup>1</sup>

Using [#11096](https://github.com/scikit-learn/scikit-learn/pull/11096)
<requires binning<sup>1</sup>



<sup>1</sup> Now, problem with binning is described in [[Bella2012]](http://dmip.webs.upv.es/papers/BFHRHandbook2010.pdf). To quote:

> The problem of using bins is that if too few bins are defined, the real probabilities are
not properly detailed to give an accurate evaluation. If too many bins are defined, the real
probabilities are not properly estimated. A partial solution to this problem is to make the bins
overlap.

And using overlapping bins seems like and additional degree of freedom, additional parameter one'd have to keep in mind, tune and argue about with collegues. 

### Additional context

I'm aware that contributors to [#11096](https://github.com/scikit-learn/scikit-learn/pull/11096) have done work in implementing Calibration loss with binning and clarifying docs on calibration topic in general. So I'd like to get feedback on that and I'm open to suggestions how to proceed.

UPD: 
Closes #18268, #21718
user: ColdTeapot273K
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_21734: state: open
title: Non-identity shrinkage targets for covariance estimation
body: ### Describe the workflow you want to enable

Discriminant analysis classification with:

1.  large number of features (`N>100`)
2. with different variances (wide range of `$\sigma^2_j$`)
3. and not too many samples (so covariance is difficult to estimate).

For `1` and `3`, the [covariance estimators](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.covariance) with shrinkage (e.g., [Ledoit-Wolf](https://scikit-learn.org/stable/modules/generated/sklearn.covariance.LedoitWolf.html) and [OAS](https://scikit-learn.org/stable/modules/generated/sklearn.covariance.OAS.html)) are quite useful. However, these assume that feature variances (`$\sigma^2_j$`) are largely the same across features. In many applications, different features may have very different variances.

This is also useful if individual features are quite noisy and mostly uncorrelated.

### Describe your proposed solution

* Add diagonal covariance estimator
    * [Dudoit et al. (2002)](https://www.tandfonline.com/doi/abs/10.1198/016214502753479248)
    * This is straightforward, simply zero-ing out the off-diagonal elements in `EmpiricalCovariance`
* Shrinkage target to diagonal covariance 
    * [Schafer and Strimmer (2005)](https://www.degruyter.com/document/doi/10.2202/1544-6115.1175/html) describes this estimator in the figure "Target D: (diagonal, unequal variance)".
    * This is implemented in R by [sparsediscrim](https://github.com/ramhiser/sparsediscrim/tree/master) and in pyx by [covar](https://github.com/rmcgibbo/covar/blob/master/covar.pyx)



### Describe alternatives you've considered, if relevant

As mentioned in https://github.com/scikit-learn/scikit-learn/issues/3508, one could standardize the data before covariance estimation and then shrink towards the identity matrix. However, per previous correspondence with Ledoit (https://github.com/scikit-learn/scikit-learn/issues/3508#issuecomment-51151623), standardizing is not strictly correct.

### Additional context

Previously discussed as a nice-to-have in https://github.com/scikit-learn/scikit-learn/issues/3508
user: charlesbmi
issue_21722: state: open
title: Improve documentation of the `impute` module
body: ### Describe the issue linked to the documentation

In #17087, it was requested to make `IterativeImputer` work with mixed types. This is useful because folks often want to impute categorical data.

In the absence of an official solution, there are a few blog posts out there as well as Stack Overflow answers doing things like creating custom `Ordinal/LabelEncoder` that preserve the missing categorical values to later apply a `KNNImputer` (see [[1]](https://www.analyticsvidhya.com/blog/2020/07/knnimputer-a-robust-way-to-impute-missing-values-using-scikit-learn/), [[2]](https://stackoverflow.com/q/64900801/554319)). Alternatively, folks resort to things like https://pypi.org/project/autoimpute/.

It seems that there are some packages that impute categorical data in R, like https://cran.r-project.org/web/packages/missMDA/index.html. However, it's not clear if there are alternatives in scikit-learn, or even in Python.

In summary, it is very difficult for a non-expert to understand if these approaches are valid, or if they lead to incoherent results for some reason.

### Suggest a potential alternative/fix

Even though #17346 was closed, it would be cool to at least document some authoritative way of how (not to) impute missing categorical data.
user: astrojuanlu
labels: Hard: Hard level of difficulty
issue_21719: state: open
title: Weights for sklearn.model_selection.KFold
body: ### Describe the workflow you want to enable

Kfold with weights. For example stratified kfold is with equal weights for all classes.

### Describe your proposed solution

Please let me know if there is such feature already available. If not, I will describe in detail.

### Describe alternatives you've considered, if relevant

Stratiied kfold can hve this as an argument

### Additional context

I need this to unequally weigh the classes in imbalanced task and generate dataset for another stage of process
user: INF800
issue_21718: state: open
title: RFC Principled metrics for scoring and calibration of supervised learning
body: We receive a lot feature requests to add new metrics for classification and regression task, each shedding light on different aspects of supervised learning, e.g. (incomplete list)
- issues
  - #26026
  - #21686
  - #21665
  - #20266
  - #20162
  - #20943
  - #18268
  - #17854
  - #15247
  - #13482
  - #10003
  - #5516
- PRs
  - #21320
  - #20877
  - #19556
  - #18970
  - #11096

Scikit-learn estimators for regression and classification give, with rare exceptions, point forecasts, i.e. a single number as prediction for a single row of input data. Point forecasts usually aim at predicting a certain property/functional of the predictive distribution such as the expectation `E[Y|X]` or the median `Median(Y|X)`. Such point forecasts are well understood and there is good theory on how to validate them, see [1].

Examples: `LinearRegression`, `Ridge`, `HistGradientBoostingRegressor(loss='squared_error')` and classifiers with a `predict_proba` like `LogisticRegression` estimate the expectation.

#### There are 2 main aspects of model validation/selection:

1. **Calibration**: How good does a model take into account the available information in form of features? Is there (conditional) bias?
  This can be assessed with identification functions, which in the case of the expectation amounts to checking for bias `sum(y_predicted - y_observed)`, see [2, 4].
2. **Scoring**: How good is the predictive performance of a model A compared to a model B? The aim is often to select the better model. This can be assessed by (consistent) scoring functions, like the (mean) squared error for the expectation `sum((y_predicted - y_observed)^2)`, see [1, 4].

### A. How to structure `metrics` in a principled way to make it clear which metric assesses what?

1. How to make the distinction between calibration and scoring more explicit?
2. How to make it clear which functional is assessed, e.g. expectation or a quantile, or 2 different quantiles?
3. What is in-scope, what is out-of-scope?

Note that (consistent) scoring functions assess calibration and resolution at the same time, see [3, 4]. They are a good fit for cross validation.

For example, #20162 proposes scoring functions and calibration functions for 2 quantiles, i.e. the prediction of 2 estimators at the same time (a 2-point forecast).

### B. How to assess calibration?

Currently, we have `calibration_curve` to assess (auto-)calibration for classifiers. It would be desirable to

1. also assess regression tasks;
2. also look at conditional calibration, i.e. aggregates grouped by some feature (either bins or categorical level);
3. handle different functionals, e.g. the expectation and quantiles.
4. What is in-scope, what is out-of-scope?

**Note** that *calibration scores*, by nature, often do not comply with "larger is better" or "smaller is better* and **are thus not suitable for cross-validation/grid search** (for hyper-parameter tuning). An example is the bias proposed in #17854, also mentioned above, which is sign sensitive (larger absolute value means worse, and the sign means under- or overshooting of the prediction).

Further examples: #11096 is for (auto-) calibration of classification only, #20162 proposes (among others) a metric for the calibration of 2 quantiles.


#### References
[1] Gneiting (2009) https://arxiv.org/abs/0912.0902
[2] Nolde & Ziegel (2016) https://arxiv.org/abs/1608.05498
[3] Pohle (2020) https://arxiv.org/abs/2005.01835
[4] Fissler, Lorentzen & Mayer (2022) https://arxiv.org/abs/2202.12780
user: lorentzenchr
issue_21706: state: open
title: fit_params for Adaboost estimator
body: ### Describe the workflow you want to enable

I am trying to use a custom estimator as the base estimator for Adaboost regressor. My custom estimator requires additional fit_params apart from sample weights. As I understand, there is no support for passing other fit_params to base_estimators via AdaBoostRegressor. 

### Describe your proposed solution

In the fit method of Adaboost, allow `**fit_params` for base_estimator. 

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: venkyyuvy
issue_21699: state: open
title: MeanThreshold, MedianThreshold, and other threshold support in GenericUnivariateSelect
body: ### Describe the workflow you want to enable

I would like to select features by thresholding their **mean** value (i.e., mean-across-samples), similar to how `VarianceThreshold` selects features by thresholding their variance-across-samples.

### Describe your proposed solution

Two possible options:
* Implement a new Estimator in `sklearn.feature_selection`, similar to `VarianceThreshold`. Example: https://github.com/hermidalc/sklearn-extensions/blob/f9296d0f3ed5d71b7f07779b47d8cf71bbcfa51b/feature_selection/_average_threshold.py#L7-L96
* Add a `mode='threshold'` option to `GenericUnivariateSelect` (https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.GenericUnivariateSelect.html)
    * This would allow the user to pass in more flexible `score_func`s as well

### Describe alternatives you've considered, if relevant

~~Another alternative, although this seems counter to how these functions are designed~~
* ~~`SelectFromModel(DummyRegressor(strategy='mean'), importance_getter='constant_', threshold=min_mean_value)`~~
    * https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html
    * https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyRegressor.html

### Additional context

Setting a MeanThreshold would be useful when working with non-negative features, such as pixel intensity in images. For example, we might want to exclude pixels that are regularly saturated in our dataset, as they may be less informative.

Specifically, in my research field of neuroscience (single-neuron recordings), our "features" are the (non-negative) action-potential-counts for each neuron. We often exclude neurons with very-low-firing-rates to minimize discretization error. Here are a few examples of neuroscience papers that set a MeanThreshold per neuron (i.e., feature):
* https://www.cell.com/neuron/pdfExtended/S0896-6273(17)30592-5
    * `Units with mean firing rates less than 1.5 Hz were excluded from the analysis.`
* https://www.nature.com/articles/s41586-021-04042-9
    * `Neurons with firing rates less than 0.5 Hz were excluded.`
* https://www.nature.com/articles/s41598-018-37227-w
    * `Firing data were Gaussian smoothed and binned in 0.1 s periods and bins with firing rates less than 0.1 Hz (no spikes) were excluded.` 
user: charlesbmi
issue_21665: state: open
title: Mean Standardized Log Loss (MSLL) for uncertainty aware regression models
body: ### Describe the workflow you want to enable

#### Why MSLL?
Traditional metrics such as [`mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html) or [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score) may not be able to correctly evaluate uncertainty aware models because they do not take predictive standard deviation (`y_std`) into account.

#### Why sklearn?
To the best of my knowledge, I am unaware of any other library in Python that is widely used and has a well-organized, standardized set of metrics implemented for ML regression models. 

#### Mean Standardized Log Loss (MSLL)
![](https://i2.paste.pics/81bcccd9205f35cc3745506dba0a7043.png)

If the above equation (standardized log loss) is averaged over all the values in `y_pred` and `y_std`, the resultant metric is Mean Standardized Log Loss (MSLL). The above equation is derived from [Eq. 2.34, pp. 23 in GPML book](http://www.gaussianprocess.org/gpml/chapters/RW.pdf) - Cited by 23541.

#### Properties
* MSLL is useful to evaluate probabilistic models (or any models that can output both `y_pred` and `y_std`, like [GaussianProcessRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html)). 
* Lower the MSLL, better the model.

#### Which algorithms from sklearn can use this metric?
* [sklearn.gaussian_process.GaussianProcessRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html)
* [sklearn.linear_model.BayesianRidge](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.BayesianRidge.html)
* [sklearn.linear_model.ARDRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ARDRegression.html)
* ...
* Maybe more

#### A demo
* An overfitted model (high MSLL)
![image](https://user-images.githubusercontent.com/59758528/141679330-5d391d8e-110d-43af-9e06-fbcfae171fa3.png)

* An underfitted model (very high MSLL)
![image](https://user-images.githubusercontent.com/59758528/141679346-b1449a4a-9abb-4300-ab79-a3142a33347f.png)

* A well-fitted model (low MSLL)
![image](https://user-images.githubusercontent.com/59758528/141679365-4ebdb2d6-c450-499b-9827-03f424cdb5f6.png)

* A comparison between RMSE and MSLL

MSLL is high if the prediction intervals are not well calibrated, on the other side, RMSE can not take this criterion into account.

![image](https://user-images.githubusercontent.com/59758528/141680846-bdf86a93-9b28-417f-9b64-7b592d15c2b2.png)

```python
# Original Author: Vincent Dubourg <vincent.dubourg@gmail.com>
#         Jake Vanderplas <vanderplas@astro.washington.edu>
#         Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>s
# License: BSD 3 clause
#
# Modified by: Zeel B Patel <patel_zeel@iitgn.ac.in>

import numpy as np
from matplotlib import pyplot as plt

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, ConstantKernel as C

np.random.seed(1)

def f(x):
    """The function to predict."""
    return x * np.sin(x)

def msll(y_true, y_pred, y_std):
    first_term = 0.5 * np.log(2 * np.pi * y_std**2)
    second_term = ((y_true - y_pred)**2)/(2 * y_std**2)
    
    return np.mean(first_term + second_term)

X = np.linspace(0.1, 9.9, 20)
X = np.atleast_2d(X).T

# Observations and noise
y = f(X).ravel()
dy = 0.5 + 1.0 * np.random.random(y.shape)
noise = np.random.normal(0, dy)
y += noise

# Instantiate a Gaussian Process model
gp = GaussianProcessRegressor(kernel=kernel, alpha=dy ** 2, n_restarts_optimizer=1)

# Fit to data using Maximum Likelihood Estimation of the parameters
gp.fit(X, y)

# Make the prediction on the meshed x-axis (ask for MSE as well)
y_pred, sigma = gp.predict(x, return_std=True)

# Plot the function, the prediction and the 95% confidence interval based on
# the MSE
plt.figure()
plt.plot(x, f(x), "r:", label=r"$f(x) = x\,\sin(x)$")
plt.errorbar(X.ravel(), y, dy, fmt="r.", markersize=10, label="Observations")
plt.plot(x, y_pred, "b-", label="Prediction")
plt.fill(
    np.concatenate([x, x[::-1]]),
    np.concatenate([y_pred - 1.9600 * sigma, (y_pred + 1.9600 * sigma)[::-1]]),
    alpha=0.5,
    fc="b",
    ec="None",
    label="95% confidence interval",
)
plt.xlabel("$x$")
plt.ylabel("$f(x)$")
plt.ylim(-10, 20)
plt.legend(loc="upper left")
plt.title('MSLL: ' + str(msll(f(x).ravel(), y_pred, sigma)))

plt.show()
```

### Describe your proposed solution

A PR to include `mean_standardized_log_loss` in `sklearn.metrics`.

#### Potential usage
```python
y_pred, y_std = model.predict(x, return_std=True)
msll = mean_standardized_log_loss(y_true, y_pred, y_std)
```

#### A rough version of `mean_standardized_log_loss` function:

```python
def mean_standardized_log_loss(
    y_true, y_pred, *, sample_weight=None, multioutput="uniform_average", squared=True
):
    """Mean standardized log loss.
    Read more in the :ref:`User Guide <mean_standardized_log_loss>`.
    Parameters
    ----------
    y_true : array-like of shape (n_samples,) or (n_samples, n_outputs)
        Ground truth (correct) target values.
    y_pred : array-like of shape (n_samples,) or (n_samples, n_outputs)
        Estimated target values.
    y_std : array-like of shape (n_samples,) or (n_samples, n_outputs)
        Estimated standard deviation in predictions.
    sample_weight : array-like of shape (n_samples,), default=None
        Sample weights.
    multioutput : {'raw_values', 'uniform_average'} or array-like of shape \
            (n_outputs,), default='uniform_average'
        Defines aggregating of multiple output values.
        Array-like value defines weights used to average errors.
        'raw_values' :
            Returns a full set of errors in case of multioutput input.
        'uniform_average' :
            Errors of all outputs are averaged with uniform weight.

    Returns
    -------
    loss : float or ndarray of floats
        A non-negative floating point value (the best value is 0.0), or an
        array of floating point values, one for each individual target.
    Examples
    --------
    >>> from sklearn.metrics import mean_standardized_log_loss
    >>> y_true = [3, -0.5, 2, 7]
    >>> y_pred = [2.5, 0.0, 2, 8]
    >>> y_std = [0.1, 0, 0.05, 0.3]
    >>> mean_standardized_log_loss(y_true, y_pred, y_std)
    6.356
    >>> y_true = [[0.5, 1],[-1, 1],[7, -6]]
    >>> y_pred = [[0, 2],[-1, 2],[8, -5]]
    >>> y_std = [[0.01, 0.02],[0.01,0.04],[0.03,0.04]]
    >>> mean_standardized_log_loss(y_true, y_pred, y_std)
    5.511
    >>> mean_squared_error(y_true, y_pred, multioutput='raw_values')
    array([5.00107605, 6.02159874])
    >>> mean_squared_error(y_true, y_pred, multioutput=[0.3, 0.7])
    2.858
    """
    # y_type, y_true, y_pred, multioutput = _check_reg_targets(
    #   y_true, y_pred, multioutput
    # )
    # check_consistent_length(y_true, y_pred, sample_weight)
    
    ###########
    # Checks like the above ones to be implemented.
    ###########
    
    first_term = 0.5 * np.log(2 * np.pi * y_std**2)
    second_term = ((y_true - y_pred)**2)/(2 * y_std**2)
    
    output_errors = np.average(first_term + second_term, axis=0, weights=sample_weight)

    if isinstance(multioutput, str):
        if multioutput == "raw_values":
            return output_errors
        elif multioutput == "uniform_average":
            # pass None as weights to np.average: uniform mean
            multioutput = None

    return np.average(output_errors, weights=multioutput)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

#### A set of publications that have used MSLL as a metric

| Publication | Venue | Citations |
| :--               | :- | --: | 
| Rasmussen, C. E. (2003, February). Gaussian processes in machine learning. In Summer school on machine learning (pp. 63-71). Springer, Berlin, Heidelberg. | | 23541 | 
| Alvarez, M. A., & Lawrence, N. D. (2011). Computationally efficient convolved multiple output Gaussian processes. The Journal of Machine Learning Research, 12, 1459-1500. | JMLR '11 | 289 |
|Chalupka, K., Williams, C. K., & Murray, I. (2013). A framework for evaluating approximation methods for Gaussian process regression. Journal of Machine Learning Research, 14, 333-350.| JMLR '13 | 144 |
| Andrew Gordon Wilson, David A. Knowles, and Zoubin Ghahramani. 2012. Gaussian process regression networks. In Proceedings of the 29th International Conference on International Conference on Machine Learning (ICML'12). Omnipress, Madison, WI, USA, 1139–1146.| ICML '12 |143 |
| Wilson, A. G., Gilboa, E., Cunningham, J. P., & Nehorai, A. (2014, December). Fast Kernel Learning for Multidimensional Pattern Extrapolation. In NIPS (pp. 3626-3634). | NIPS '14 | 141 |
| Chen, Z., & Wang, B. (2018). How priors of initial hyperparameters affect Gaussian process regression models. Neurocomputing, 275, 1702-1710. | Neurocomputing '18 | 44 |
| Nguyen, D. T., Filippone, M., & Michiardi, P. (2019, April). Exact gaussian process regression with distributed computations. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing (pp. 1286-1295). |  | 8 |
|Lederer, A., Conejo, A. J. O., Maier, K. A., Xiao, W., Umlauft, J., & Hirche, S. (2021, July). Gaussian Process-Based Real-Time Learning for Safety Critical Applications. In International Conference on Machine Learning (pp. 6055-6064). PMLR. | ICML '21 | 2 |
| Lederer, A., Conejo, A. J. O., Maier, K. A., Xiao, W., Umlauft, J., & Hirche, S. (2021, July). Gaussian Process-Based Real-Time Learning for Safety Critical Applications. In International Conference on Machine Learning (pp. 6055-6064). PMLR. | ICML '21 |2 |
| Gümbel, S., & Schmidt, T. (2020). Machine learning for multiple yield curve markets: fast calibration in the Gaussian affine framework. Risks, 8(2), 50. | | 1 |

#### Acknowledgement

I would like to thank @wesselb for suggesting this metric for the first time to me.
user: patel-zeel
issue_21639: state: open
title: Default Value for SimpleImputer with all missing values
body: ### Describe the workflow you want to enable

In the _Notes_ of the `SimpleImputer`, it says that:

> Columns which only contained missing values at fit are discarded upon transform if strategy is not "constant".

It'd be nice to be able to set this to a default value in `transform` instead of dropping those columns if there is no valid summary statistic in `fit`. 

I recognize that it may only be a couple of lines, but it seems useful to have the logic captured within the class instead of checking beforehand for all `np.nan`s per column. 

### Describe your proposed solution

Near the lines https://github.com/scikit-learn/scikit-learn/blob/74bf394b872d9ed9a91a949f04dbcf2239c558e7/sklearn/impute/_base.py#L512-L525

We could add something like 

```python
            invalid_mask = _get_mask(statistics, np.nan)
            valid_mask = np.logical_not(invalid_mask)
            valid_statistics = statistics[valid_mask]
            valid_statistics_indexes = np.flatnonzero(valid_mask)

            invalid_statistics_indexes = np.flatnonzero(invalid_mask) 
            ...
            X[:, invalid_statistics_indexes] = {default_value}
            # X = X[:, valid_statistics_indexes]
```


### Describe alternatives you've considered, if relevant

An alternative is to do this logic before running the SimpleImputer. 

### Additional context

_No response_
user: vsocrates
issue_21599: state: open
title: `_check_feature_names` raises UserWarning when accessing bagged estimators
body: ### Describe the bug

Using the estimators inside a `BaggingRegressor` to predict data raises a 

> UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names

coming from `_check_feature_names`, even if the `BaggingRegressor` or its base estimator (in this case `DecisionTreeRegressor`) are able to take the feature names into account while fitting.

### Steps/Code to Reproduce

``` python
from sklearn.ensemble import BaggingRegressor
import pandas as pd
import numpy as np

df = pd.DataFrame(
    {
        "feature_name": [-12.32, 1.43, 30.01, 22.17],
        "target": [72, 55, 32, 43],
    }
)
X = df[["feature_name"]]
y = df["target"]

bagged_trees = BaggingRegressor()
bagged_trees.fit(X, y)
bagged_trees_predictions = bagged_trees.predict(X) # rises no warning
bagged_trees.estimators_[0].predict(X) # rises UserWarning
```

### Expected Results

No warning should be thrown

### Actual Results

/home/arturoamor/miniforge3/envs/scikit-learn-course/lib/python3.9/site-packages/sklearn/base.py:438: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
  warnings.warn(
array([72., 55., 32., 32.])

### Versions

System:
    python: 3.9.5 | packaged by conda-forge | (default, Jun 19 2021, 00:32:32)  [GCC 9.3.0]
executable: /home/arturoamor/miniforge3/envs/scikit-learn-course/bin/python
   machine: Linux-5.13.0-1017-oem-x86_64-with-glibc2.31

Python dependencies:
          pip: 21.1.3
   setuptools: 49.6.0.post20210108
      sklearn: 1.0.1
        numpy: 1.21.0
        scipy: 1.7.0
       Cython: None
       pandas: 1.3.0
   matplotlib: 3.4.2
       joblib: 1.0.1
threadpoolctl: 2.1.0
user: ArturoAmorQ
issue_21571: state: open
title: add graph-based clustering
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Fixes #21570


#### What does this implement/fix? Explain your changes.

**Graph-Based Clustering** ([original repo link](https://github.com/dayyass/graph-based-clustering)) using connected components and minimum spanning trees.
Both suggested clustering methods are transductive - meaning they are not designed to be applied to new, unseen data.

#### Any other comments?

All information in the reference issue.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: dayyass
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_21570: state: open
title: add graph-based clustering
body: ### Describe the workflow you want to enable

#### Graph-Based Clustering ([original repo link](https://github.com/dayyass/graph-based-clustering))

**Graph-Based Clustering** using connected components and minimum spanning trees.
Both suggested clustering methods are **transductive** - meaning they are not designed to be applied to new, unseen data.

#### ConnectedComponentsClustering

This method computes pairwise distances matrix on the input data, and using *threshold* (parameter provided by the user) to binarize pairwise distances matrix makes an undirected graph in order to find connected components to perform the clustering.

#### SpanTreeConnectedComponentsClustering

This method computes pairwise distances matrix on the input data, builds a graph on the obtained matrix, finds minimum spanning tree, and finaly, performs the clustering through dividing the graph into *n_clusters* (parameter given by the user) by removing *n-1* edges with the highest weights.

### Describe your proposed solution

#### ConnectedComponentsClustering

Required arguments:
- **threshold** - paremeter to binarize pairwise distances matrix and make undirected graph

Optional arguments:
- **metric** - sklearn.metrics.[pairwise_distances](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise_distances.html) parameter (default: *"euclidean"*)
- **n_jobs** - sklearn.metrics.[pairwise_distances](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise_distances.html) parameter (default: *None*)

Example:

```python3
import numpy as np
from graph_based_clustering import ConnectedComponentsClustering

X = np.array([[0, 1], [1, 0], [1, 1]])

clustering = ConnectedComponentsClustering(
    threshold=0.275,
    metric="euclidean",
    n_jobs=-1,
)

clustering.fit(X)
labels_pred = clustering.labels_

# alternative
labels_pred = clustering.fit_predict(X)
```

#### SpanTreeConnectedComponentsClustering

Required arguments:
- **n_clusters** - the number of clusters to find

Optional arguments:
- **metric** - sklearn.metrics.[pairwise_distances](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise_distances.html) parameter (default: *"euclidean"*)
- **n_jobs** - sklearn.metrics.[pairwise_distances](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise_distances.html) parameter (default: *None*)

Example:

```python3
import numpy as np
from graph_based_clustering import SpanTreeConnectedComponentsClustering

X = np.array([[0, 1], [1, 0], [1, 1]])

clustering = SpanTreeConnectedComponentsClustering(
    n_clusters=3,
    metric="euclidean",
    n_jobs=-1,
)

clustering.fit(X)
labels_pred = clustering.labels_

# alternative
labels_pred = clustering.fit_predict(X)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

#### ConnectedComponentsClustering

![image](https://user-images.githubusercontent.com/26326659/140605364-6871b5a2-9022-462a-acb5-2f1541395d28.png)<br/>

#### SpanTreeConnectedComponentsClustering

![image](https://user-images.githubusercontent.com/26326659/140605377-d76da44f-aace-401c-80aa-f51c5eca847a.png)<br/>
user: dayyass
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_21553: state: open
title: Cross endianness and bitness pickle issues with KNeighborsClassifier / KDTree
body: Reported in https://github.com/scikit-learn/scikit-learn/issues/21237 (cross endianness). There is a similar issue for the cross bitness, to reproduce:

Generate a pickle on a 64bit machine:
```py
from sklearn.datasets import make_classification
X, y = make_classification(random_state=0)

from sklearn.neighbors import KNeighborsClassifier
clf = KNeighborsClassifier(algorithm='kd_tree')
clf.fit(X, y)
import pickle
pickle.dump(clf, open('/tmp/kneighbors.pkl', 'wb'))
```

Open it on a 32bit machine:
```
docker run -it -v /tmp:/io lesteve/i386-scikit-learn python3 -c 'import pickle; pickle.load(open("/io/kneighbors.pkl", "rb"))'
```
Output:
```
WARNING: The requested image's platform (linux/386) does not match the detected host platform (linux/amd64) and no specific platform was requested
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "sklearn/neighbors/_binary_tree.pxi", line 1062, in sklearn.neighbors._kd_tree.BinaryTree.__setstate__
    self._update_memviews()
  File "sklearn/neighbors/_binary_tree.pxi", line 1004, in sklearn.neighbors._kd_tree.BinaryTree._update_memviews
    self.idx_array = self.idx_array_arr
ValueError: Buffer dtype mismatch, expected 'ITYPE_t' but got 'long long'
```

Quite likely solving the issue is rather similar to https://github.com/scikit-learn/scikit-learn/pull/21552 and https://github.com/scikit-learn/scikit-learn/pull/21539.
user: lesteve
issue_21543: state: open
title: permutation_test_score shuffling when a group enforces a specific label
body: ### Describe the bug

I believe there is a bug or design expectation misunderstanding with `permutation_test_score` label shuffle function code. Maybe there is a use case for only shuffling within groups, but to me the default use case should be to shuffle all the samples while ensuring all samples within a group have the same shuffled label. This would be the equivalent behavior as if no groups existed, i.e. each label was in it's own unique group, which is the way I expected it to work.

### Steps/Code to Reproduce

```python
from sklearn.model_selection._validation import _shuffle
from sklearn.utils import check_random_state

y = [1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
groups = [1, 2, 3, 4, 4, 5, 6, 7, 7, 8]

random_state = check_random_state(0)
for _ in range(10):
    print(_shuffle(y, groups, random_state), "\n", groups, "\n", sep="")

print()

# this should work the same as if groups=None but it doesn't
groups = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
for _ in range(10):
    print(_shuffle(y, groups, random_state), "\n", groups, "\n", sep="")
```

### Expected Results

Labels should be shuffled even when there are label groups. Label groups are to make sure all labels in that group simply stay together during CV shuffling and I expected that behavior here too. The labels for a group should be treated like a single sample where they get shuffled together with another sample label group.




### Actual Results

**No labels get shuffled**:

```python
[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 4, 5, 6, 7, 7, 8]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 4, 5, 6, 7, 7, 8]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 4, 5, 6, 7, 7, 8]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 4, 5, 6, 7, 7, 8]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 4, 5, 6, 7, 7, 8]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 4, 5, 6, 7, 7, 8]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 4, 5, 6, 7, 7, 8]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 4, 5, 6, 7, 7, 8]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 4, 5, 6, 7, 7, 8]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 4, 5, 6, 7, 7, 8]


[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

[1, 0, 0, 1, 1, 0, 1, 0, 0, 1]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
```

### Versions

1.0.1 and all previous versions with `permutation_test_score` (I looked at source code it hasn't changed)

I specifically ran my test on:

System:
    python: 3.8.10 | packaged by conda-forge | (default, May 11 2021, 07:01:05)  [GCC 9.3.0]
executable: /home/hermidalc/soft/miniconda3/envs/sklearn-bio-workflows-r36/bin/python
   machine: Linux-5.14.14-200.fc34.x86_64-x86_64-with-glibc2.10

Python dependencies:
       pip: 21.1.2
setuptools: 49.6.0.post20210108
   sklearn: 0.22.2.post1
     numpy: 1.21.0
     scipy: 1.5.3
    Cython: None
    pandas: 1.2.5
matplotlib: 3.4.2
    joblib: 1.0.1

Built with OpenMP: True


user: hermidalc
issue_21504: state: open
title: TST Check correct interactions of `class_weight` and `sample_weight`
body: In scikit-learn, some estimators support `class_weight` and `sample_weight`.

It might be worth testing the correct interaction of those two types of weights, especially asserting that:
 - setting a class weights to zero is equivalent to excluding the samples associated to this class from the calibration even when using non uniform sample weights.
 - setting some samples weights to zero is equivalent to excluding those samples from the calibration even when if they are associated to using non uniform class weights.

---

**Relevant interfaces:**
 - [ ] the main subclasses of `sklearn.tree.BaseDecisionTree` for classification, i.e.:
   - [ ] `sklearn.tree.DecisionTreeClassifier`
   - [ ] `sklearn.tree.ExtraTreeClassifier`
 - [ ] the main subclasses of `sklearn.ensemble.BaseForest` for classification and embedding, i.e.:
   - [ ] `sklearn.ensemble.RandomTreesEmbedding`
   - [ ] `sklearn.ensemble.RandomForestClassifier`
   - [ ] `sklearn.ensemble.ExtraTreesClassifier`
 - [ ] `sklearn.linear_model.LogisticRegression`
 - [ ] `sklearn.linear_model.LogisticRegressionCV`
 - [ ] `sklearn.CalibratedClassifierCV` after the merge of #17541
user: jjerphan
labels: Moderate: Anything that requires some knowledge of conventions and best practices
module:test-suite: everything related to our tests
Meta-issue: General issue associated to an identified list of tasks
issue_21490: state: open
title: [DOC] Outdated description of attributes oob_decision_function_ and oob_prediction_ in bagging estimators
body: ### Describe the issue linked to the documentation

This issue concerns:
* [sklearn.ensemble.RandomForestClassifier](https://scikit-learn.org/dev/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier)
* [sklearn.ensemble.RandomForestRegressor](https://scikit-learn.org/dev/modules/generated/sklearn.ensemble.RandomForestRegressor.html#sklearn.ensemble.RandomForestRegressor)
and probably:
* [sklearn.ensemble.ExtraTreesClassifier](https://scikit-learn.org/dev/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html#sklearn.ensemble.ExtraTreesClassifier)
* [sklearn.ensemble.ExtraTreesRegressor](https://scikit-learn.org/dev/modules/generated/sklearn.ensemble.ExtraTreesRegressor.html#sklearn.ensemble.ExtraTreesRegressor)
* [sklearn.ensemble.BaggingClassifier](https://scikit-learn.org/dev/modules/generated/sklearn.ensemble.BaggingClassifier.html#sklearn.ensemble.BaggingClassifier)
* [sklearn.ensemble.BaggingRegressor](https://scikit-learn.org/dev/modules/generated/sklearn.ensemble.BaggingRegressor.html#sklearn.ensemble.BaggingRegressor)

The documentation of `oob_decision_function_` in RandomForestClassifier is:
> Decision function computed with out-of-bag estimate on the training set. If n_estimators is small it might be possible that a data point was never left out during the bootstrap. In this case, oob_decision_function_ might contain NaN. This attribute exists only when oob_score is True.

The description of `oob_prediction_` RandomForestRegressor is:
> Prediction computed with out-of-bag estimate on the training set. This attribute exists only when oob_score is True.

However, PR #19162 changed the computation of OOB scores and NaN have been replaced with 0. The description in RandomForestRegressor does not even mention the issue that a data point could never be left out.

The PR also dropped support for multiclass-multioutput so I'm not sure if the shape of `oob_decision_function_` may still be `(n_samples, n_classes, n_outputs)` (not 100% sure on this one).

### Suggest a potential alternative/fix

For random forest estimators, the new descriptions could be something like this:
```
oob_decision_function_ : ndarray of shape (n_samples, n_classes)
        Decision function computed with out-of-bag estimate on the training
        set. If ``n_estimators`` is small it might be possible that a data point
        was never left out during the bootstrap. In this case, the probabilities
        for such data points to belong to each class are 0. This attribute exists
        only when ``oob_score`` is True.
```
and
```
oob_prediction_ : ndarray of shape (n_samples,) or (n_samples, n_outputs)
       Prediction computed with out-of-bag estimate on the training set. If 
       ``n_estimators`` is small it might be possible that a data point was never
       left out during the bootstrap. In this case, the predictions for such data points
       are 0. This attribute exists only when ``oob_score`` is True.
```

The descriptions for the other estimators may also require some updates.
user: johannfaouzi
issue_21484: state: open
title: There is no list of regressors that support multioutput regression natively
body: ### Describe the issue linked to the documentation

In the [multi-output ](https://scikit-learn.org/stable/modules/multiclass.html)section of the documentation, there is a comprehensive list of which classifiers support multi-class classification. There is no such list for multi-output regression.

### Suggest a potential alternative/fix

Provide a list of regressors that support multi-output regression natively
user: eschibli
issue_21458: state: open
title: ENH Added vocabulary_count_ attribute to CountVectorizer 
body: #### What does this implement/fix? Explain your changes.

Added an attribute called `vocabulary_count_` to the CountVectorizer, which is a dictionary mapping the vocabulary terms to a count of their usage (as seen during `fit` or `fit_transform`)

user: caallinson
labels: Needs Decision - API: Requires decision regarding API
issue_21452: state: open
title: ColumnTransformers should use get_feature_names_out() when columns attribute is not available
body: ### Describe the bug

Now that transformers have a `get_feature_names_out()` method, `ColumnTransformer`s should make use of it when column names are lost in a pipeline.

### Steps/Code to Reproduce

Suppose we have a DataFrame with 3 columns A, B, C. We want to use a SimpleImputer for both A and B, followed by a StandardScaler for A and a MinMaxScaler for B. And we want a OneHotEncoder for C.

A direct mapping from this description to pipelines and column transformers looks like this (see this [gist](https://colab.research.google.com/gist/ageron/94157a5e94ec565b9e9b8a9065f1c275/scikit-learn-issue-columntransformer-and-get_feature_names_out.ipynb)):

```python
import numpy as np
import pandas as pd
from sklearn.compose import make_column_transformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder, MinMaxScaler

df = pd.DataFrame({"A": [1.,2.,np.nan], 
                   "B": [4.,np.nan,6.],
                   "C": ["G1", "G2", "G1"]})

num_pipeline = make_pipeline(
    SimpleImputer(),
    make_column_transformer(
        (StandardScaler(), ["A"]),
        (MinMaxScaler(), ["B"]),
    )
)

preprocessing = make_column_transformer(
    (num_pipeline, ["A", "B"]),
    (OneHotEncoder(), ["C"]),
)

X = preprocessing.fit_transform(df)
```

### Expected Results

I expect this code to work, since all the required information is available to the `Pipeline` and `ColumnTransformer`s: they could use `get_feature_names_out()` whenever they cannot rely on the `columns` attribute.

### Actual Results

The code above raises an `AttributeError`:

**Stacktrace:**

<details>

```stacktrace
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/sklearn/utils/__init__.py in _get_column_indices(X, key)
    407         try:
--> 408             all_columns = X.columns
    409         except AttributeError:

AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
15 frames
<ipython-input-2-ef089cab1885> in <module>()
     23 )
     24 
---> 25 X = preprocessing.fit_transform(df)

/usr/local/lib/python3.7/dist-packages/sklearn/compose/_column_transformer.py in fit_transform(self, X, y)
    673         self._validate_remainder(X)
    674 
--> 675         result = self._fit_transform(X, y, _fit_transform_one)
    676 
    677         if not result:

/usr/local/lib/python3.7/dist-packages/sklearn/compose/_column_transformer.py in _fit_transform(self, X, y, func, fitted, column_as_strings)
    613                     message=self._log_message(name, idx, len(transformers)),
    614                 )
--> 615                 for idx, (name, trans, column, weight) in enumerate(transformers, 1)
    616             )
    617         except ValueError as e:

/usr/local/lib/python3.7/dist-packages/joblib/parallel.py in __call__(self, iterable)
   1039             # remaining jobs.
   1040             self._iterating = False
-> 1041             if self.dispatch_one_batch(iterator):
   1042                 self._iterating = self._original_iterator is not None
   1043 

/usr/local/lib/python3.7/dist-packages/joblib/parallel.py in dispatch_one_batch(self, iterator)
    857                 return False
    858             else:
--> 859                 self._dispatch(tasks)
    860                 return True
    861 

/usr/local/lib/python3.7/dist-packages/joblib/parallel.py in _dispatch(self, batch)
    775         with self._lock:
    776             job_idx = len(self._jobs)
--> 777             job = self._backend.apply_async(batch, callback=cb)
    778             # A job can complete so quickly than its callback is
    779             # called before we get here, causing self._jobs to

/usr/local/lib/python3.7/dist-packages/joblib/_parallel_backends.py in apply_async(self, func, callback)
    206     def apply_async(self, func, callback=None):
    207         """Schedule a func to be run"""
--> 208         result = ImmediateResult(func)
    209         if callback:
    210             callback(result)

/usr/local/lib/python3.7/dist-packages/joblib/_parallel_backends.py in __init__(self, batch)
    570         # Don't delay the application, to avoid keeping the input
    571         # arguments in memory
--> 572         self.results = batch()
    573 
    574     def get(self):

/usr/local/lib/python3.7/dist-packages/joblib/parallel.py in __call__(self)
    261         with parallel_backend(self._backend, n_jobs=self._n_jobs):
    262             return [func(*args, **kwargs)
--> 263                     for func, args, kwargs in self.items]
    264 
    265     def __reduce__(self):

/usr/local/lib/python3.7/dist-packages/joblib/parallel.py in <listcomp>(.0)
    261         with parallel_backend(self._backend, n_jobs=self._n_jobs):
    262             return [func(*args, **kwargs)
--> 263                     for func, args, kwargs in self.items]
    264 
    265     def __reduce__(self):

/usr/local/lib/python3.7/dist-packages/sklearn/utils/fixes.py in __call__(self, *args, **kwargs)
    207     def __call__(self, *args, **kwargs):
    208         with config_context(**self.config):
--> 209             return self.function(*args, **kwargs)
    210 
    211 

/usr/local/lib/python3.7/dist-packages/sklearn/pipeline.py in _fit_transform_one(transformer, X, y, weight, message_clsname, message, **fit_params)
    889     with _print_elapsed_time(message_clsname, message):
    890         if hasattr(transformer, "fit_transform"):
--> 891             res = transformer.fit_transform(X, y, **fit_params)
    892         else:
    893             res = transformer.fit(X, y, **fit_params).transform(X)

/usr/local/lib/python3.7/dist-packages/sklearn/pipeline.py in fit_transform(self, X, y, **fit_params)
    432             fit_params_last_step = fit_params_steps[self.steps[-1][0]]
    433             if hasattr(last_step, "fit_transform"):
--> 434                 return last_step.fit_transform(Xt, y, **fit_params_last_step)
    435             else:
    436                 return last_step.fit(Xt, y, **fit_params_last_step).transform(Xt)

/usr/local/lib/python3.7/dist-packages/sklearn/compose/_column_transformer.py in fit_transform(self, X, y)
    670         self._check_n_features(X, reset=True)
    671         self._validate_transformers()
--> 672         self._validate_column_callables(X)
    673         self._validate_remainder(X)
    674 

/usr/local/lib/python3.7/dist-packages/sklearn/compose/_column_transformer.py in _validate_column_callables(self, X)
    350                 columns = columns(X)
    351             all_columns.append(columns)
--> 352             transformer_to_input_indices[name] = _get_column_indices(X, columns)
    353 
    354         self._columns = all_columns

/usr/local/lib/python3.7/dist-packages/sklearn/utils/__init__.py in _get_column_indices(X, key)
    409         except AttributeError:
    410             raise ValueError(
--> 411                 "Specifying the columns using strings is only "
    412                 "supported for pandas DataFrames"
    413             )

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

</details>

### Versions

System:
    python: 3.7.12 (default, Sep 10 2021, 00:21:48)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0
        numpy: 1.19.5
        scipy: 1.4.1
       Cython: 0.29.24
       pandas: 1.1.5
   matplotlib: 3.2.2
       joblib: 1.0.1
threadpoolctl: 3.0.0

Built with OpenMP: True

user: ageron
issue_21447: state: open
title: Ridge with cholesky solver returns NaN
body: ### Describe the bug

When using Ridge with a big number of features and the Cholesky solver, the model gets full of NaNs.
Playing with the number of features we can see that the imprecision grows until it becomes NaN.

The bug is architecture-dependent.
The provided snippet works perfectly in an `Intel(R) Xeon(R) CPU E3-1585L v5 @ 3.00GHz`.
It fails when running in an `Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz`.

A workaround was found by blindly exporting OPENBLAS_CORETYPE="Broadwell".

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.linear_model import Ridge

# we need something big to reproduce:
n_inputs = 1000
n_targets = 1
n_samples = 20000

rng = np.random.default_rng(seed=42)

b = rng.random((n_targets,))
w = rng.random((n_targets, n_inputs)) / n_inputs

X = rng.random((n_samples, n_inputs))
y = rng.random((n_samples, n_targets))

for i in range(n_samples):
    y[i] = (w @ X[i]) + b

# "lsqr" does not have the same problem as "cholesky"
model = Ridge(alpha=0.0, tol=1e-6, solver="cholesky")
model.fit(X, y)

np.testing.assert_almost_equal(model.coef_, w)
np.testing.assert_almost_equal(model.intercept_, b)

error = np.abs(model.predict(X) - y).sum()
np.testing.assert_almost_equal(error, 0.0, decimal=4)

assert np.isfinite(model.coef_).all()
assert np.isfinite(model.intercept_).all()

print("done!")
```

### Expected Results

done!

### Actual Results

Numpy assertion errors as `model.coef_` is full of `NaN`.

### Versions

The software is the same on both machines:
```
System:
    python: 3.8.9 (default, Apr  2 2021, 11:20:07)  [GCC 10.3.0]
executable: ***/.venv/bin/python
   machine: Linux-3.10.0-1062.9.1.el7.x86_64-x86_64-with-glibc2.2.5

Python dependencies:
          pip: 20.2.3
   setuptools: 54.2.0.post0
      sklearn: 0.24.1
        numpy: 1.20.2
        scipy: 1.6.1
       Cython: 0.29.22
       pandas: 1.2.3
   matplotlib: 3.4.1
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```
user: joao-m-afonso
issue_21446: state: open
title: Use a stable sorting algorithm when selecting the `max_features` in TfidfVectorizer
body: ### Describe the workflow you want to enable

Currently when selecting `max_features` in TfidfVectorizers, the algorithm (line [1172](https://github.com/scikit-learn/scikit-learn/blob/844b4be24d20fc42cc13b957374c718956a0db39/sklearn/feature_extraction/text.py#L1172)) uses [numpy.argsort](https://numpy.org/doc/stable/reference/generated/numpy.argsort.html) with the `quicksort` enabled  (the default argument).

Given that quicksort is not stable, this leads to inconsistent behavior across different datasets, for example:

```python 
corpus = ['AAA', 'AAB', 'AAC', 'AAD', 'AAE', 'AAF']
vectorizer = TfidfVectorizer(smooth_idf=False, max_features=5)
vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names_out())   # ['aaa' 'aab' 'aac' 'aad' 'aae']
```
As all the features have the same frequency they are return in lexicographic order, this does not happens for the following dataset:
```python 
corpus = ['AAA', 'AAB', 'AAC', 'AAD', 'AAE', 'AAF', 'AAG', 'ABA', 'ABB', 'ABC', 'ABD', 'ACA', 'ACB', 'ADA', 'AEA',
          'AFA', 'BAA']
vectorizer = TfidfVectorizer(smooth_idf=False, max_features=5)
vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names_out())  # ['aaa' 'aca' 'acb' 'ada' 'aea']
```

I would expect the result to be the same as the first example but is not. 

### Describe your proposed solution

Since `1.15.0` the option stable was added to numpy.argsort, so simply changing the line to:

```python 
mask_inds = (-tfs[mask]).argsort(kind="stable")[:limit]
```

should be enough to solve the problem. 

### Describe alternatives you've considered, if relevant

As an alternative use the option `mergesort` that is stable

```python 
mask_inds = (-tfs[mask]).argsort(kind="mergesort")[:limit]
```

### Additional context

```
sklearn.__version__
'1.0'
```
user: mesejo
issue_21415: state: open
title: Inequality Constrained Quantile Regression
body: ### Describe the workflow you want to enable

Exciting times to be a quantile regressor in python! I have spent quite a while trying to port the R quantreg package into python. The authors of the R package are also the authors of the method on which the current QuantileRegressor is based. One feature in quantreg that should be easy to implement  into QuantileRegressor  is inequality constrained quantile regression. A common use-case is as follows: you know from first principles that some independent variable should be positively correlated with the dependent variable; you therefore want to constrain your model coefficient to be >= 0. 

Implementing inequality constrained quantile regression as a linear program has a relatively straightforward solution - see https://www.jstor.org/stable/25053440 (cited by 94, published 2005). This solution has been implemented in the popular R package - see https://cran.r-project.org/web/packages/quantreg/index.html (cited by 851). Full disclosure - I need this use case in order to incorporate QuantileRegressor into my day to day.

### Describe your proposed solution

I propose QuantileRegressor is modified by adding two properties: R and r. R and r are both of type Option[np.ndarray]. According to the notation in both the aforementioned paper and R package, model coefficients _theta_ are constrained such that R _theta_ >= r. The call to linprog would need to be modified to optionally include parameters A_ub and b_ub, which can be calculated from R and r.

### Describe alternatives you've considered, if relevant

I'm not thrilled about the `r` and `R` properties being added to QuantileRegressor. I would have much preferred to use something like `Bounds`. Unfortunately, its not clear how we could apply both an upper and lower bound to multiple coefficients.

We could also have a separate class called ConstrainedQuantileRegressor. However, there would be a ton of overlap between the fit methods with only a small difference in the call to linprog. 

### Additional context

I would very much like to be the person to work on this. I think I have a pretty good idea on how it could be implemented on the maths side.
user: kaftand
labels: Needs Decision - Include Feature: Requires decision regarding including feature
Needs Investigation: Issue requires investigation
issue_21410: state: open
title: Using a custom rbf kernel function for sklearn's SVC is way faster than built-in method
body: ### Describe the bug

I've noticed a rather strange behavior when using **Scikit-Learn's SVC** implementation. Using the **built-in rbf kernel** with SVC is **slower** by magnitudes than passing a **custom rbf function to SVC()**. 

From what I could see and understand so far, the only difference between the two versions is that **in the built-in rbf case, not sklearn but libsvm will compute the kernel**. Passing a dedicated kernel function as hyperparameter to SVC() leads to the computation of the kernel inside sklearn, not in libsvm. The results are identical, but **the latter case takes only a fraction of the computation time**. 

I've created a toy dataset that mimics the data I am currently working on. This probably only becomes relevant with larger datasets. There, the discrepancy in computation time likely increases.

Does anyone know why this is happening? Is this the expected behavior?


### Steps/Code to Reproduce
```py
import numpy as np
from time import time
from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.metrics.pairwise import rbf_kernel
from sklearn.metrics import accuracy_score

# create toy data
n_features = 1000
n_samples = 10000
n_informative = 10
X, y = make_classification(n_samples, n_features, n_informative=n_informative)
gamma = 1 / n_features

# fit SVC with built-in rbf kernel
svc_built_in = SVC(kernel='rbf', gamma=gamma)
np.random.seed(13)
t1 = time()
svc_built_in.fit(X, y)
acc = accuracy_score(y, svc_built_in.predict(X))
print("Fitting SVC with built-in kernel took {:.1f} seconds".format(time()-t1))
print("Accuracy: {}".format(acc))

# fit SVC with custom rbf kernel
svc_custom = SVC(kernel=rbf_kernel, gamma=gamma)
np.random.seed(13)
t1 = time()
svc_custom.fit(X, y)
acc = accuracy_score(y, svc_custom.predict(X))
print("Fitting SVC with a custom kernel took {:.1f} seconds".format(time()-t1))
print("Accuracy: {}".format(acc))
```
### Expected Results

I would have assumed that both versions should run in the same time.

### Actual Results

Fitting SVC with built-in kernel took 58.6 seconds
Accuracy: 0.9846
Fitting SVC with a custom kernel took 3.2 seconds
Accuracy: 0.9846

### Versions

System:
    python: 3.8.12 (default, Oct 12 2021, 13:49:34)  [GCC 7.5.0]
executable: /home/nwinter/anaconda3/envs/mmll_gists/bin/python
   machine: Linux-5.11.0-37-generic-x86_64-with-glibc2.17

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 1.0
        numpy: 1.21.3
        scipy: 1.7.1
       Cython: None
       pandas: 1.3.4
   matplotlib: 3.4.3
       joblib: 1.1.0
threadpoolctl: 3.0.0

Built with OpenMP: True
user: NilsWinter
issue_21409: state: open
title: [feat] enable the n_jobs for mutual info regression and classifier
body: Before, mutal_info use NearestNeighbors but did not enable n_jobs to accelerate the speed
user: bingoko
issue_21394: state: open
title: [Feature Request] Add support for negative coefficient constraints
body: Is it possible to enhance the scikit-learn - linear models by replacing the **positive** parameter with a **bounds** parameter which will take a list of bounds to constrain the coefficients to?
user: rohan-gt
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_21391: state: open
title: add sklearn.metrics Display class to plot Precision/Recall/F1 for probability thresholds
body: ### Describe the workflow you want to enable

Working with binary classifiers I often, in addition to PR-curve and ROC-curve, need Precision / Recall / F1 (y-axis) for probability thresholds (x-axis).

### Describe your proposed solution

```python
import numpy as np
from sklearn.metrics import precision_recall_curve, PrecisionRecallF1Display

y_true = np.array([0, 0, 1, 1])
y_scores = np.array([0.1, 0.4, 0.35, 0.8])

precision, recall, thresholds = precision_recall_curve(y_true, y_scores)

display = PrecisionRecallF1Display(precision, recall, thresholds, plot_f1=True)
display.plot()
```

<img width="392" alt="prf1-curve" src="https://user-images.githubusercontent.com/26326659/138347199-9059cfd8-cbbd-4f62-b055-5160b7e94f70.png">



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: dayyass
issue_21388: state: open
title: Allow inspect.PartialDependenceDisplay to work with a prediction function
body: ### Describe the workflow you want to enable

Currently, partial dependence plots (PDP) and ICE profiles can be conveniently visualized by calling `PartialDependenceDisplay.from_estimator`. This works as long as the model is a valid scikit-learn estimator. 

Since PDP/ICE actually only require the availability of a prediction function, I'd love to be able to pass only a prediction function taking an array-like input X and providing numeric output values.

### Describe your proposed solution

Add method `PartialDependenceDisplay.from_predict_fun` that would enable above solution.

Trivial (but not working) example:

```python
import numpy as np
from sklearn.inspection import PartialDependenceDisplay

X = np.array([range(10)]).reshape(5, -1)

def predict_fun(X):
    return np.repeat(3, X.shape[0])

PartialDependenceDisplay.from_predict_fun(
    predict_fun,
    X
)
```

### Describe alternatives you've considered, if relevant

Define custom estimator.

### Additional context

_No response_
user: mayer79
issue_21377: state: open
title: Skewed Gaussian Process Regressor
body: ### Describe the workflow you want to enable

Allow fitting of a multivariate lognormal distribution over the data set.



### Describe your proposed solution

By the training fit for each point a lognormal distribution parameterized with `shape`, `scale` and `location`.
The kernel needs to be adapted according to [1] and [2].
By prediction return the `shape`, `scale` and  `location`.


### Describe alternatives you've considered, if relevant

For one dimensional y and constant "skewness" over X it is sufficient to logarithmically scale y and fit a normal GPR.
This can be done in the preprocessing of the data, the scaling factor of the column can be added to theta.

### Additional context

[1] Gaussian process regression with skewed errors
December 2019Journal of Computational and Applied Mathematics 
DOI: 10.1016/j.cam.2019.112665.

[2] Skew Gaussian processes for classification
Vol.:(0123456789)Machine Learning (2020) 109:1877–1902
https://doi.org/10.1007/s10994-020-05906-3
user: yuvalyitz
issue_21356: state: open
title: SequentialFeatureSelector.fit() groups parameter
body: ### Describe the workflow you want to enable

`SequentialFeatureSelector` allows specifying a cross-validation splitter via the `cv` parameter in the constructor, but does not allow the `groups` parameter in `fit()` so group-based cross-validation cannot be used.

### Describe your proposed solution

Add a `groups` parameter to `SequentialFeatureSelector.fit()` to allow groups to be passed into the cross-validation splitter:
https://github.com/scikit-learn/scikit-learn/blob/2ec028d2e8c34e332c851311e8cf330128081a1c/sklearn/feature_selection/_sequential.py#L143

This will need to be passed through to `_get_best_new_feature()` and then to `cross_val_score()`.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: agkphysics
issue_21354: state: open
title: [MRG] Add alt text to scikit-learn documentation
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Addresses #21214. (I'm not using `fixes` because this does not cover the entire issue and I don't think it should be closed.) 

#### What does this implement/fix? Explain your changes.

This PR adds [alt text](https://moz.com/learn/seo/alt-text) to some of scikit-learn's documentation. It includes alt text for
- all images on [2.4 Biclustering](https://scikit-learn.org/stable/modules/biclustering.html)
- all images on [1.16 Probability calibration](https://scikit-learn.org/stable/modules/calibration.html)
- some images on [2.3 Clustering](https://scikit-learn.org/stable/modules/clustering.html)
- all images on [2.6 Covariance estimation](https://scikit-learn.org/stable/modules/covariance.html)

#### Any other comments?

This alt text was contributed during an [alt text writing event](https://www.meetup.com/PyLadies-SWFL/events/280761048/) by some wonderful volunteers, so this PR does not intend to cover adding alt text to all of scikit-learn's documentation.

As far as I'm aware, alt text is a new kind of contribution to this project. I'm happy to help discuss what this might include in terms of maintenance and/or review in the future if that is helpful.

Thanks in advance for reviewing this PR!

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: isabela-pf
issue_21320: state: open
title: [MRG] FEA Lift metric and curve
body: ### Reference Issues/PRs

implements partially what is in two stale PRs in a significantly more complete manner:
* #18479
* #10003

### What does this implement/fix? Explain your changes.

Implemented the **lift_score** metric function, with the **lift_curve** function and the **LiftCurveDisplay** class.

Lift is a commonly used metric in evaluating response to ad campaigns. please read:
* https://en.wikipedia.org/wiki/Lift_(data_mining)
* https://www.ibm.com/docs/en/spss-statistics/24.0.0?topic=overtraining-cumulative-gains-lift-charts

This implementation includes:

* implementation of lift_score in ``sklearn/metrics/_classification.py`` for calculating a single value
* Implementation of lift_curve in ``sklearn/metrics/_ranking.py`` for calculating an array of lifts based on different positive classification rates
* Implementation of LiftCurveDisplay in ``sklearn/metrics/_plot/lift_curve.py`` for plotting the lift curve/chart
* Added documentation in the model_evaluation section for lift and referenced the function page which contains examples

user: nawarhalabi
labels: Needs Decision: Requires decision
issue_21319: state: open
title: Heteroskedastacity-Aware Covariance estimators
body: ### Describe the workflow you want to enable

Scikit-learn provides multiple covariance estimators useful for different purposes. Heteroskedastacity-Aware Estimators are designed for correcting bias due to autocorrelation in data, and thus are extremely useful when estimating correlation between time series. To the best of my understanding, such an estimator is not currently available

### Describe your proposed solution

Implement covariance Andrews and Newey-West estimators
https://www.jstor.org/stable/pdf/2938229.pdf
https://en.wikipedia.org/wiki/Newey%E2%80%93West_estimator

### Describe alternatives you've considered, if relevant

To some extent, the methods are implemented in statsmodels package

### Additional context

_No response_
user: aleksejs-fomins
issue_21313: state: open
title: Improve make_scorer for multiclass classification, or add new metrics to "common cases" predefined values
body: ### Describe the workflow you want to enable

I am trying to run multiclass classification evaluation and compare it w/ some experiments with binary classification. The table in https://scikit-learn.org/stable/modules/model_evaluation.html for "common cases" doesn't contain all the updated metrics, like `Cohen_kappa_score`, which would be nice to add.

In addition, Cohen kappa score and balanced accuracy state they work with multiclass classification. However, they require one to have 1D arrays for both `y` and `y_pred`. This results in me having to define another lambda function for them.

Here X = (100,5), and y=(100, 3) (so 3 classes)
```
# some scorers need to get defined with this for multiclass
def cohen_scorer(clf, X, y):
    y_pred = clf.predict(X).argmax(axis=1)
    y = y.argmax(axis=1)
    return cohen_kappa_score(y, y_pred)

def balanced_accuracy_scorer(clf, X, y):
    y_pred = clf.predict(X).argmax(axis=1)
    y = y.argmax(axis=1)
    return balanced_accuracy_score(y, y_pred)

multiclass_scoring_funcs = {
    'balanced_accuracy': balanced_accuracy_scorer,
    'cohen_kappa_score': cohen_scorer,
    'roc_auc_ovr': 'roc_auc_ovr',# roc_ovr,
    'f1_micro': 'f1_micro', #f1_micro,
    'recall_micro': 'recall_micro',
    'precision_micro': 'precision_micro',
    'jaccard_micro': 'jaccard_micro',
}

# run cross-validation
scores = cross_validate(
    clf,
    X,
    y,
    groups=subjects,
    cv=cv,
    scoring=scoring,
    return_estimator=True,
    return_train_score=True,
    n_jobs=-1,
    error_score='raise'
)
```

### Describe your proposed solution

 Ideally, they can just do `y.argmax(axis=1)` if they encounter a 2D array within `cohen_kappa_score` and `balanced_accuracy_score`, or even more general, let `make_scorer` know?

That way, if e.g. `cohen_kappa_score` doesn't work out of the box for multi-class classification, then `make_scorer(take_argmax=True,...` will just pass `y.argmax(axis=1)` and `y_pred.argmax(axis=1)`. This would then make setting up the scoring objects for `cross_validate` easier, because you can just rely on `make_scorer` instead of defining a new function.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: adam2392
issue_21312: state: open
title: Performance of the RBF kernel in epsilon-SVR (SVM)
body: ### Describe the bug

My simple implementation of the RBF kernel is significantly faster than the default implementation for `n_samples << n_features`. How does this happen? The effect diminishes if `n_samples >> n_features`, in this case both implementations have a similar runtime.

### Steps/Code to Reproduce

[Gist `n_samples << n_features`](https://gist.github.com/Turakar/a45b836311467dd75d779b7c1cbdb5ac)
[Gist `n_samples >> n_features`](https://gist.github.com/Turakar/aa531f2fcc93cb9064b4e6690d98c965)

### Expected Results

The default C implementation is faster than a naive implementation in python.

### Actual Results

The naive implementation in python is 10x faster than the default C implementation for `n_samples << n_features`.

### Versions

```
System:
    python: 3.8.12 (default, Aug 30 2021, 00:00:00)  [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]
executable: /home/t/.cache/pypoetry/virtualenvs/mutation-prediction-VhT0dLh3-py3.8/bin/python
   machine: Linux-5.14.10-200.fc34.x86_64-x86_64-with-glibc2.2.5

Python dependencies:
          pip: 21.0.1
   setuptools: 54.1.2
      sklearn: 0.24.2
        numpy: 1.21.0
        scipy: 1.7.0
       Cython: None
       pandas: 1.2.5
   matplotlib: 3.4.2
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```
I ran the benchmark on a Ryzen 7 2700 Octa-Core (Hyperthreading enabled) with 64 GB RAM.
user: Turakar
issue_21311: state: open
title: Fix column_transformer to use fitparams like Pipeline
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes #19465.



#### What does this implement/fix? Explain your changes.
This fix implements fitparams in the same way as Pipelines to meet parity


#### Any other comments?


user: jschmidtml
issue_21305: state: open
title: LogisticRegression with SAGA using sample_weight does not converge
body: ### Describe the bug

I am fitting a logistic regression model on a sparse matrix and a binary response. Many of the rows of the matrix are repeated, so to speed things up I switched to a smaller sparse matrix with non-repeated rows and use the repetitions of the rows to calculate the `sample_weight` argument to `fit`.

The issue is that when I work with the weighted model, `fit` produces a warning that it dit not converge because it took too many steps. 

I looked a bit under the hood of `fit` and `sag_solver` does some scaling of `alpha` and `beta` using `n_samples`. The resulting `alpha_scaled` and `beta_scaled` are different between the weighted and unweighted cases and they should not be (the loss function is the same). Perhaps the equivalent scaling for the weighted case should be the sum of the weights (if the intended 'unit' of the weights is 'count') and not just `n_samples`. Not sure if this is the issue, but it just it made me worry that the `sample_weight` argument is used in a bit naive way just as a multiplier for the loss function, while there might be scaling implications that are not accounted for when deciding when to stop. 

UPDATE: It seems that the issue is with the SAGA solver. I tried with liblinear and it seems to work. This will solve my immediate problem, because for now I only want the L1-reg. It is still good to look into this, because at the moment only SAGA offers elasticnet.

### Steps/Code to Reproduce

[vttrifonov/logistic_sample_weights.ipynb](https://gist.github.com/vttrifonov/55ded32d11f93d26e64fdb2d4c7434d2)

### Expected Results

In the above code I expect the second fit to run much faster than the first and to produce the same coefficients. 

### Actual Results

It fails in both.

### Versions

System:
    python: 3.7.0 (default, Jun 28 2018, 07:39:16)  [Clang 4.0.1 (tags/RELEASE_401/final)]
executable: /Users/vtrifonov/projects/tiny-proteins/env/bin/python
   machine: Darwin-19.6.0-x86_64-i386-64bit

Python dependencies:
          pip: 21.2.2
   setuptools: 58.0.4
      sklearn: 0.24.2
        numpy: 1.20.3
        scipy: 1.7.1
       Cython: None
       pandas: 1.3.2
   matplotlib: 3.4.2
       joblib: 1.0.1
threadpoolctl: 2.2.0

Built with OpenMP: True
user: vttrifonov
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_21292: state: open
title: Add partial model evaluation to gradient boosting
body: This PR is an enhancement proposal and it does not relate to any PR.

#### What does this implement/fix?

I found myself wanting to evaluate partially-trained GB models in the `monitor` function. This turned out to be impossible since some of the estimators in `self.estimators_` were not there yet, and `None`'s were being passed on to `predict_stages` (in `\estimators\_gradient_boosting.pyx`). This PR adds a None-check to allow for this.

#### Comments 

I've played around with having the None-check on the cython level, but I've decided it would be better to keep the inner loops tight. Also, note the comment in `_staged_raw_predict`.
user: MarcinKonowalczyk
issue_21291: state: open
title: Another input needed for the parameter `n_features_to_select` in SequentialFeatureSelector
body: ### Describe the workflow you want to enable

Currently, to use the SequentialFeatureSelector, you need a parameter `n_features_to_select`.  However, according to the book 'Introduction to statistical learning', you can know how many number of variables are appropriate only after you test all number of parameters and get the best model of each number of parameters. 

![image](https://user-images.githubusercontent.com/34431729/136651763-fb27c479-1dee-46ad-aadb-90ecdc23dd42.png)

This is an excerpt from the book ISLR, which shows that you can figure out how many number of features are appropriate after testing all numbers of predictors. You cannot figure it out beforehand.

### Describe your proposed solution

I suggest to create options like "the lowest adjusted r_squared" for the parameter `n_features_to_select`. By doing so, you can choose the number of features that has the lowest adjusted r_squared, which cannot be known beforehand.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: hellojinwoo
issue_21290: state: open
title: HTML representation of estimators: center the diagrams
body: ### Describe the issue linked to the documentation

The HTML representation of the pipeline is currently left-centered.

### Suggest fix
Center the diagram.

### References
#18305

### Suggest a potential alternative/fix

_No response_
user: reshamas
labels: Needs Decision: Requires decision
issue_21280: state: open
title: Make it possible to pass an arbitrary probablistic classifier as `method` for CalibratedClassifierCV
body: ### Describe the workflow you want to enable

In addition to `method="sigmoid"` and `method="isotonic"` it would be great to pass any scikit-learn compatible classifier with a `predict_proba` to `CalibratedClassifierCV`.

In particular I would like to be able to pass: `method=HistGradientBoostingClassifier(monotonic_cst=[1])` (or #13649) to calibrate a model using a non-parametric method (similar to isotonic regression) but with an adjustable overfitting/underfitting tradeoff by setting `max_leaf_nodes` and `max_iter` (which is not possible with isotonic calibration).

### Describe your proposed solution

The `sklearn.calibration._fit_calibrator` private helper would need to accept and clone scikit-learn regressors instead of raising `ValueError`.

Of course one would need to add new tests + extend at least one of the calibration example to demonstrate this new capability.


Edit: the first version of this issue used `HistGradientBoostingRegressor` instead of `HistGradientBoostingClassifier`
user: ogrisel
issue_21274: state: open
title: SpectralClustering breaks normalized Laplacian when the row sums are negative
body: ### Describe the bug

_laplacian_dense from scipy\sparse\csgraph\_laplacian.py breaks, crushing SpectralClustering


### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.cluster import SpectralClustering
x = np.random.randn(100, 10)
x = x @ x.T
spectral = SpectralClustering(n_clusters=4, affinity='precomputed',
    random_state=0).fit(x)
result = list(spectral.labels_)
print(result)
```

### Expected Results

No error

### Actual Results

```python-traceback
$ C:/Users/Name/AppData/Local/Programs/Python/Python37/python.exe c:/Users/Name/Downloads/tmp1.py
C:\Users\Name\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\sparse\csgraph\_laplacian.py:118: RuntimeWarning: invalid value encountered in sqrt
  w = np.where(isolated_node_mask, 1, np.sqrt(w))
Traceback (most recent call last):
  File "c:/Users/Name/Downloads/tmp1.py", line 6, in <module>
    random_state=0).fit(x)
  File "C:\Users\Name\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\cluster\_spectral.py", line 590, in fit
    verbose=self.verbose,
  File "C:\Users\Name\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\cluster\_spectral.py", line 308, in spectral_clustering
    drop_first=False,
  File "C:\Users\Name\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\manifold\_spectral_embedding.py", line 300, in spectral_embedding
    laplacian, k=n_components, sigma=1.0, which="LM", tol=eigen_tol, v0=v0
  File "C:\Users\Name\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\sparse\linalg\eigen\arpack\arpack.py", line 1645, in eigsh
    hermitian=True, tol=tol)
  File "C:\Users\Name\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\sparse\linalg\eigen\arpack\arpack.py", line 1069, in get_OPinv_matvec
    return LuInv(A).matvec
  File "C:\Users\Name\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\sparse\linalg\eigen\arpack\arpack.py", line 936, in __init__
    self.M_lu = lu_factor(M)
  File "C:\Users\Name\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\linalg\decomp_lu.py", line 
70, in lu_factor
    a1 = asarray_chkfinite(a)
  File "C:\Users\Name\AppData\Local\Programs\Python\Python37\lib\site-packages\numpy\lib\function_base.py", line 489, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs
```

### Versions

System:
    python: 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\Name\AppData\Local\Programs\Python\Python37\python.exe
   machine: Windows-10-10.0.19041-SP0

Python dependencies:
          pip: 21.2.4
   setuptools: 58.1.0
      sklearn: 1.0
        numpy: 1.21.2
        scipy: 1.7.1
       Cython: 0.29.24
       pandas: 1.3.3
   matplotlib: 3.4.3
       joblib: 1.0.1
threadpoolctl: 2.2.0
user: lobpcg
issue_21266: state: open
title: Display all parameter values in a tabular for a tab of the notebook HTML repr of estimators
body: ### Describe the workflow you want to enable

I would to make it easy to quickly introspect the default hyperparameter values of scikit-learn estimator in a jupyter notebook session.

### Describe your proposed solution

The HTML widget returned by `BaseEstimator._repr_mimebundle_` could display a tab with a tabular rendering of the content of the `estimator.get_params()` method.

In the case of meta-estimators with complex diagrams, we could optionally have an option to display this only for a subestimator of interest but I have no particular UI suggestion in mind.

### Additional context

Loosely related to #21240 which is about another improvement, this time for the text rendering of complex estimators.
user: ogrisel
issue_21240: state: open
title: Improve _EstimatorPrettyPrinter to use an indentation style a bit more consistent with black
body: The current indentation style used in `_EstimatorPrettyPrinter` is too horizontal and suboptimal rendering of complex pipelines and column transformers in the notebook HTML widgets. See for instance:

https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html#html-representation-of-pipeline

![image](https://user-images.githubusercontent.com/89061/135991404-1396a8d1-cf43-4010-b9da-93eb2700fbc9.png)

I think using a more vertical indentation style would help (and would furthermore be more consistent with the black formatting we use in our code base):

For this specific example black might over-do it a bit (see below) although I still find it better than what we currently have.

```python
Pipeline(
    steps=[
        (
            "preprocessor",
            ColumnTransformer(
                transformers=[
                    (
                        "num",
                        Pipeline(
                            steps=[
                                ("imputer", SimpleImputer(strategy="median")),
                                ("scaler", StandardScaler()),
                            ]
                        ),
                        ["age", "fare"],
                    ),
                    (
                        "cat",
                        OneHotEncoder(handle_unknown="ignore"),
                        ["embarked", "sex", "pclass"],
                    ),
                ]
            ),
        ),
        ("classifier", LogisticRegression()),
    ]
)
```

Or maybe we could find a middle ground?
user: ogrisel
issue_21225: state: open
title: Added support for sampling for exponential, linear and epanechnikov kernels in KernelDensity
body: Also fixed existing tests and added more tests for the sampling. Also updated the current implementation of uniform sampling for consistency


#### Reference Issues/PRs

Fixes #21224

#### What does this implement/fix? Explain your changes.

Adds support for `KernelDensity.sample` for the `linear`, `exponential` and `epanechnikov` kernels. `KernelDensity.sample` creates new points from the fit distribution, which is done by sampling a random point from the training dataset and adding a sample from the kernel distribution. I have added tests for the new kernels and have adjusted documentation where necessary.

user: lrjball
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_21224: state: open
title: Kernel Density adding .sample() support for linear and exponential kernels
body: ### Describe the workflow you want to enable

Currently `KernelDensity.sample` only works for the `gaussian` and `tophat` kernels, but I'd like to add support for the `linear` and `exponential` kernels as well. 

### Describe your proposed solution

The sampling process amounts to picking a point from the original dataset at random, then sampling from the kernel distribution and adding it to that point. The `laplace` distribution from numpy can be used for the `exponential` kernel and the `triangular` distribution from numpy can be used to generate samples from the `linear` kernel, so this change is only adding a few extra lines, plus some tests. I have a working version for this so will put in a PR.

Adding for `linear` and `exponential` for now as they are the easiest to implement. `epanechnikov` and `cosine` are possible as well, but are slightly more involved.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: lrjball
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_21214: state: open
title: Add alt text to scikit-learn documentation 
body: ## Describe the issue linked to the documentation
Adding alt text to images permits visually impaired users to have greater access.

## Suggest a potential alternative/fix

## About [Alt Text](https://moz.com/learn/seo/alt-text)

Alt text (alternative text), also known as "alt attributes," “alt descriptions," or technically incorrectly as "alt tags,” are used within an HTML code to describe the appearance and function of an image on a page.

Alt text uses:
1. Adding alternative text to photos is first and foremost a principle of web accessibility. Visually impaired users using screen readers will be read an alt attribute to better understand an on-page image.

2. Alt text will be displayed in place of an image if an image file cannot be loaded.

3. Alt text provide better image context/descriptions to search engine crawlers, helping them to index an image properly.


## Reference
- https://github.com/isabela-pf/scikit-learn/pull/1
- [Chartability](https://chartability.fizz.studio)
- [Meetup event](https://www.meetup.com/PyLadies-SWFL/events/280761048/) PyLadies SWFL & Miami, Python SWFL
  - Event Video:  [Make open source accessible: Isabela Presedo-Floyd and Mars Lee](https://youtu.be/dDpimPYOKuc) 

## Questions
1.  For scikit-learn, what is the maximum line length for writing alt text descriptions for images?
2. Is there a way to do a `grep` of the library and see how many images exist in the documentation?
3. Can you confirm that the images are static?  They are produced from code, within the documentation.  Are images always the same that are produced?

user: reshamas
issue_21211: state: open
title: ENH use cv_results in the different curve display to add confidence intervals
body: This PR intends to add the capability of plotting uncertainty of the different curves (calibration, precision-recall, roc, etc.) by using the results of cross-validation (i.e. the output of `cross_validate`).

## TODO:

- [ ] add a parameter `return_indices` in `cross_validate` to store the train-test indices. It is the safest way to keep track of the train-test splits in the case of stochastic splitting strategies.
- [ ] add a method `from_cv_results` in the plotting display to take advantage of the CV computation.
- [ ] add unit test for `from_cv_results`
- [ ] add unit test for the new keyword parameters in `CalibrationDisplay`
- [ ] add unit test for the new strategy of binning in `calibration_curve`

## Usage example

```python
# %%
import numpy as np
from sklearn.datasets import make_classification

X, y = make_classification(
    n_samples=10_000, weights=[0.1, 0.9], random_state=42, class_sep=1
)
sample_weight = np.zeros_like(y, dtype=np.float64)
sample_weight[y == 0] = 0.1
sample_weight[y == 1] = 0.9

# %%
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test, sw_train, sw_test = train_test_split(
    X, y, sample_weight, random_state=42
)

# %%
from sklearn.linear_model import LogisticRegression
from sklearn.calibration import CalibratedClassifierCV

calibration_method = "isotonic"
models = {
    "LR no weights": LogisticRegression(),
    "LR class weights": LogisticRegression(class_weight="balanced"),
    "Calibrated LR no weights": CalibratedClassifierCV(
        LogisticRegression(),
        method=calibration_method,
    ),
    "Calibrated LR class weights": CalibratedClassifierCV(
        LogisticRegression(class_weight="balanced"),
        method=calibration_method,
    ),
    "Calibrated LR sample weights": CalibratedClassifierCV(
        LogisticRegression(),
        method=calibration_method,
    ),
    "Calibrated LR class and sample weights": CalibratedClassifierCV(
        LogisticRegression(class_weight="balanced"),
        method=calibration_method,
    ),
}

# %%
import matplotlib.pyplot as plt
from sklearn.calibration import CalibrationDisplay
from sklearn.metrics import balanced_accuracy_score

fig, ax = plt.subplots()

calibration_display_params = {
    "n_bins": 20,
    "strategy": "quantile",
}
for name, model in models.items():
    if "sample weights" in name:
        model.fit(X_train, y_train, sample_weight=sw_train)
    else:
        model.fit(X_train, y_train)

    score = balanced_accuracy_score(y_test, model.predict(X_test))
    CalibrationDisplay.from_estimator(
        model,
        X_test,
        y_test,
        name=name + f" - {score:.3f}",
        ax=ax,
        **calibration_display_params,
    )
ax.legend(loc="center left", bbox_to_anchor=(1, 0.5), title="Model - Balanced Accuracy")
_ = fig.suptitle(f"Using {calibration_method} calibration")

# %%
from sklearn.model_selection import cross_validate
from sklearn.model_selection import KFold

cv_results = {}
cv = KFold(n_splits=5)
for name, model in models.items():
    if "sample weights" in name:
        fit_params = {"sample_weight": sample_weight}
    else:
        fit_params = {}
    cv_results[name] = cross_validate(
        model,
        X,
        y,
        cv=cv,
        fit_params=fit_params,
        scoring="balanced_accuracy",
        return_estimator=True,
        return_indices=True,
    )

# %%
fig, ax = plt.subplots()
for model_idx, (name, results) in enumerate(cv_results.items()):
    CalibrationDisplay.from_cv_results(
        results, X, y, ax=ax, name=name, **calibration_display_params
    )
ax.legend(loc="center left", bbox_to_anchor=(1, 0.5))

# %%
fig, ax = plt.subplots()
for model_idx, (name, results) in enumerate(cv_results.items()):
    CalibrationDisplay.from_cv_results(
        results,
        X,
        y,
        ax=ax,
        name=name,
        plot_uncertainty_style="fill_between",
        **calibration_display_params,
    )
ax.legend(loc="center left", bbox_to_anchor=(1, 0.5))

# %%
fig, ax = plt.subplots()
for model_idx, (name, results) in enumerate(cv_results.items()):
    CalibrationDisplay.from_cv_results(
        results,
        X,
        y,
        ax=ax,
        name=name,
        plot_uncertainty_style="lines",
        **calibration_display_params,
    )
ax.legend(loc="center left", bbox_to_anchor=(1, 0.5))

# %%
```

![001](https://user-images.githubusercontent.com/7454015/135686762-30fc2c65-4671-48de-bb35-405008411d37.png)
![002](https://user-images.githubusercontent.com/7454015/135686764-a2674e40-7050-4f1f-950d-f2f5deaaf19d.png)
![003](https://user-images.githubusercontent.com/7454015/135686767-cd57bbb0-aa36-4ba0-b097-9a4f0a99e632.png)
![004](https://user-images.githubusercontent.com/7454015/135686768-b22c1db9-81f2-428a-85c0-2f8cc8e9a8b8.png)
user: glemaitre
issue_21209: state: open
title: Support `dualtree=True` (or other kwargs) in `NearestNeighbors.kneighbors()`
body: ### Describe the workflow you want to enable

`BallTree` and `KDTree` support `dualtree=True` (and other keyword arguments, such as `breadth_first`) in their `.query()` method which can provide significant speedups in some use cases. However, they do not provide out-of-the-box parallel processing support. On the other hand, the generic `NearestNeighbors` class supports parallel processing with `n_jobs>1`, but its `.kneighbors()` method seems to never call the underlying (KD- or ball-) tree with dualtree enabled.

So currently, we can either
- use  `NearestNeighbors` which supports parallel processing but not dualtree, or we can
- use `BallTree` or `KDTree` directly for dualtree support but without parallel processing,

but not enjoy both performance improvements at the same time.


### Describe your proposed solution

A quick solution would be to add the `dualtree` (and `breadth_first`) parameter (or more generically, a `**kwargs` argument) to `NearestNeighbors.kneighbors()` and pass that through to the underlying `tree.query()` function.
In fact, according to https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neighbors/_base.py, it may be sufficient to pass it through to `_tree_query_parallel_helper()`, which already supports keyword arguments and passes them on to the `.query()` method of the tree.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: bwegge
issue_21207: state: open
title: CountVectorizer(lowercase=True,strip_accents='unicode') may produce vocab that contains uppercase chars
body: ### Describe the bug

Some characters are transformed to uppercase chars (eg ™ -> TM), despite lowercase=True.
This then gives warning messages "UserWarning: Upper case characters found in vocabulary while 'lowercase' is True"
(OK, there are worse things happening in the world ;-). Thanks for scikit-learn)

### Steps/Code to Reproduce

```python
from sklearn.feature_extraction.text import CountVectorizer
x = ['This is Problematic™.','THIS IS NOT']

cv = CountVectorizer(
    lowercase=True,
    strip_accents='unicode',
    ngram_range=(1,1)
    )

x_v = cv.fit_transform(x)
print(cv.get_feature_names_out()) # Contains "problematicTM"

# then you can get a warning
# "UserWarning: Upper case characters found in vocabulary while 'lowercase' is True"
# if you both create a classifier AND run cv.fit_transform
# (no message if you only call one)

from sklearn.svm import LinearSVC
y = [1,0]
xtest = ['This is not']
ytest = [0]

# comment either of the 2 following lines, no warning message displayed
clf = LinearSVC(random_state=42).fit(x_v, y)
xtest_v = cv.transform(xtest)
```

### Expected Results

print: ['is' 'not' 'problematictm' 'this']

### Actual Results

print: ['is' 'not' 'problematicTM' 'this']

warning message:
/Users/fps/_fps/DeveloperTools/virtualenvs/fps_env/lib/python3.9/site-packages/sklearn/feature_extraction/text.py:1208: UserWarning: Upper case characters found in vocabulary while 'lowercase' is True. These entries will not be matched with any documents
  warnings.warn(

### Versions

/Users/fps/_fps/DeveloperTools/virtualenvs/fps_env/lib/python3.9/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
  warnings.warn(

System:
    python: 3.9.2 (v3.9.2:1a79785e3e, Feb 19 2021, 09:06:10)  [Clang 6.0 (clang-600.0.57)]
executable: /Users/fps/_fps/DeveloperTools/virtualenvs/fps_env/bin/python3
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 21.2.4
   setuptools: 49.2.1
      sklearn: 1.0
        numpy: 1.21.2
        scipy: 1.7.1
       Cython: None
       pandas: 1.3.3
   matplotlib: None
       joblib: 1.0.1
threadpoolctl: 2.2.0

Built with OpenMP: True
user: fpservant
issue_21170: state: open
title: FEA Add variable importance to linear models
body: ### Describe the workflow you want to enable

I'd like to have a feature importance method native to linear models (without L1 penalty) that is calculated on the training set:
```python
clf = LogisticRegression(with_importance=True)
clf.fit(X, y)
clf.feature_importances_  # or some nice plot thereof
```

### Describe your proposed solution
#### New proposal
Evaluate if the LMG (Lindeman,  Merenda  and  Gold, see [1, 2]) is applicable and feasible for L2 penalized regression and for GLMs. Else, consider other measures of [1, 2].

In short, LMG is Shapley value decomposition of R2 by the features.

**References:**
- [1] R package [relaimpo](https://cran.r-project.org/package=relaimpo) with JSS paper [U. Grömping (2006). Relative Importance for Linear Regression in R: The Package relaimpo](http://www.jstatsoft.org/v17/i01)
- [2] [U. Grömping (2016). Variable importance in regression models](https://prof.bht-berlin.de/fileadmin/prof/groemp/downloads/Gr%c3%b6mping2015_WiresCS_AcceptedVersion.pdf)

#### Original proposal
Compute the t-statistic of the coefficients
```
t[j] = coef[j] / std(coef[j])
```
and use the absolute, i.e. `|t|`, as measure of (in-sample)  importance. For GLMs like the logistic regression, see section 5.3 in https://arxiv.org/pdf/1509.09169.pdf for a formula of `Var[coef]`.

### Describe alternatives you've considered, if relevant

Any general importance measure (permutation importance, SHAP values, ...) also works.

### Additional context

Given the great and legitimate need for interpretability, I would favor to have a native importance measure for linear models. Random Forests have their own native `feature_importances_` with the warning
> impurity-based feature importances can be misleading for high cardinality features (many unique values).

We could add a similar warning for collinear features like
> feature importances can be misleading for collinear or high-dimensional features.

I guess, in the end, this is true for all feature importance measures, even for SHAP (see also our [multicollinear example](https://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance_multicollinear.html#sphx-glr-auto-examples-inspection-plot-permutation-importance-multicollinear-py)).

Prior discussions like #16802, #6773,  #13048, focued on p-values which seem out-of-scope for scikit-learn for different reasons. I hope we can circumvent these reasons by focusing on feature importance only and not considering p-values.

user: lorentzenchr
labels: Needs Decision: Requires decision
issue_21138: state: open
title: Example with ranking metrices
body: ### Describe the issue linked to the documentation

Some of the metrices in #2805 were implemented in #7739.

### Suggest a potential alternative/fix

It would be nice to add (to) an example the usage of those ranking metrices with the addition of, e.g., kendall's tau and spearman's rho:
```python
from scipy.stats import kendalltau, spearmanr
from sklearn.metrics import make_scorer

kenall_tau_score = make_scorer(kendalltau)
spearman_rho_score = make_scorer(kendalltau)
```
user: lorentzenchr
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_21132: state: open
title: ENH add new common test: `test_negative_sample_weight_support` and `allow_negative_sample_weight` tag 
body: #### Reference Issues/PRs
Fixes #15531.

#### What does this implement/fix? Explain your changes.
A new tag is added: `allow_negative_sample_weight`. By default it is True, and it is changed to False if the estimator doesnt
supports negative `sample_weight`. In this case there is a check for the negative case. This far the following estimators
checked for negative weights: `LinearRegression`, `AdaBoostClassifier`, `AdaBoostRegressor`, and `KDEDensity`. 
<br>
Some estimators raised misleading error in the negative weight case: `BayesianRidge`,  `KernelRidge`,  `Ridge`, `RidgeClassifier`, `ElasticNetCV` and `LassoCV`. These are updated. Some estimators which uses another estimator copies its allow_negative_sample_weight tag. It is needed because those estimators fit method will be used, so they have to handle sample_weights the same way: `RANSACRegressor`, `StackingRegressor`, `StackingClassifier`, `VotingRegressor`, `VotingClassifier` and `MultiOutputEstimator`.
<br>
A common test is added: `test_negative_sample_weight_support`. This checks all estimators in `_tested_estimators()` where the estimator has `sample_weight` parameter in `fit`. If the estimator has False tag then the appropriate error is expected, else no error. 
<br>
A helper function is added: `_create_data_for_fit`. This function is needed to run the `fit` method in the test.

#### Any other comments?
Now every estimator in `_tested_estimators()` is explicitly tested by using neagative weights in fit and check for result. There is no estimator which uses `sample_weight` in `fit` and it is not listed in  `_tested_estimators()`. The changes are backward compatible, because only those cases were updated where the negative weight case raised another error so far.


user: simonandras
labels: Needs Decision: Requires decision
module:test-suite: everything related to our tests
issue_21058: state: open
title: ENH HTML repr shows best estimator in *SearchCV when refit=True
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Closes https://github.com/scikit-learn/scikit-learn/issues/20971

#### What does this implement/fix? Explain your changes.
Show the `best_estimator_` if the search refits.

#### Any other comments?

Here is a simple test if you want to see it locally:

```python
from sklearn.model_selection import GridSearchCV
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import make_classification
from sklearn import set_config

set_config(display='diagram')

X, y = make_classification(random_state=0)

pipeline = Pipeline([("scale", StandardScaler()), ("clf", DummyClassifier())])
search_cv = GridSearchCV(
    pipeline,
    param_grid={"clf": [DummyClassifier(), LogisticRegression()]},
)
search_cv.fit(X, y)
```
Will show:

<img width="205" alt="Screen Shot 2021-09-15 at 4 19 46 PM" src="https://user-images.githubusercontent.com/5402633/133503712-362223aa-df14-4073-bab7-86f5dc046631.png">



<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
issue_21042: state: open
title: Specify Priors in Multiclass Classification
body: ### Describe the workflow you want to enable

Currently there doesn't appear to be a way to specify priors in GaussianNB() in a pipeline with BinaryRelevance or OneVsRest because you can only specify priors for one label. Would be great if there were a workaround.

Issue discussed in a little more detail here.

https://stackoverflow.com/questions/50556191/specify-priors-in-multi-label-naive-bayes-in-python-scikit-learn

### Describe your proposed solution

Perhaps passing an array of arrays into binaryrelevance somehow with priors?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: npschweizer
issue_21033: state: open
title: Add custom_range argument for partial dependence
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #20890 

#### What does this implement/fix? Explain your changes.
This PR allows users to specify a `custom_range` of values to calculate partial depedency for some or all of the `features`.

The api is `custom_range={feature: array-like of grid values}`. 

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: freddyaboulton
issue_21010: state: open
title: Adding a reset weights strategy to Adaboost Regressor
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
#20443
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
I added a resetting weights strategy when an error rate increases during training. I implemented it as @ogrisel suggested 
in https://github.com/scikit-learn/scikit-learn/issues/20443#issuecomment-914621599 .

I also added a test code, but it takes some time to finish.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: europeanplaice
issue_21007: state: open
title: Feature request: confusion matrix could report confidence intervals
body: ### Describe the workflow you want to enable

Motivation: https://arxiv.org/pdf/2006.11105.pdf

The values reported for the TP, FN, FP, TN rates are only estimates having an uncertainty.
It would be great if sklearn.metrics.confusion_matrix could also return the estimates for the uncertainty (this is especially important for small numbers of classifications).

I guess as a start it would be enough to provide a confidence interval using the normal approximation or the F distribution, see "Exact Binomial Confidence Interval
for Proportions" Jeffrey T. Morisette and Siamak Khorram  https://www.researchgate.net/profile/Jeffrey-Morisette-2/publication/255630317_Exact_Binomial_Confidence_Interval_for_Proportions/links/5601ba7708aecb0ce881aea9/Exact-Binomial-Confidence-Interval-for-Proportions.pdf 

The confidence intervals would help to assess the quality of a classifier.

### Describe your proposed solution

sklearn.metrics.confusion_matrix could optionally return a matrix with the (half width?) of the (potentially) asymmetric confidence intervals. Potentially, the 95% confidence level would be a good default.

### Describe alternatives you've considered, if relevant

The user could bootstrap confidence intervals on his own via bootstrapping the standard error.

```python
import numpy as np
from scipy.stats import bernoulli
from sklearn.metrics import confusion_matrix

#generate random examples for computing the confusion matrix
p=0.5
y_true = bernoulli.rvs(p, size=100)
y_pred = bernoulli.rvs(p, size=100)
cm = confusion_matrix(y_true, y_pred, normalize = "true")
print("non-bootsrapped confidence matrix", cm)

data = np.vstack((y_true,y_pred)).T
def bootstrap(data, statistic, boostrap_size, num_boostrap_samples=1000):
    indices=range(len(data))
    confusion_matrices=[]
    for i in range(num_boostrap_samples):
        choice= np.random.choice(indices, size = boostrap_size, replace = True)
        y_true = data[choice,0]
        y_pred = data[choice,1]
        cm = statistic(y_true, y_pred, normalize = "true")
        confusion_matrices.append(cm)
    return np.average(confusion_matrices, axis = 0), np.std(confusion_matrices, axis = 0)
cm, standard_error_of_confidence_matrix= bootstrap(data, confusion_matrix, boostrap_size= len(data))
    
print("bootsrapped confidence matrix", cm)
print("bootsrapped standard error of confidence matrix", standard_error_of_confidence_matrix)
```

which outputs for 
```
non-bootsrapped confidence matrix 
[[0.53488372 0.46511628]
 [0.43859649 0.56140351]]
bootsrapped confidence matrix
 [[0.53863312 0.46136688]
 [0.44174818 0.55825182]]
bootsrapped standard error of confidence matrix 
[[0.07670171 0.07670171]
 [0.0663557  0.0663557 ]]
```
### Additional context

_No response_
user: jonaslandsgesell
issue_21006: state: open
title: Possibly biased KNeighborsClassifier for ties
body: ### Describe the bug

Hello team scikit-learn,

On investigating `KNeighborsClassifier` it seems that for an even number of neighbors, in case of a tie, the classifier incorrectly assigns a label to the outcome with a lower numerical value.

i.e. for a binary classification problem, if the labels are `0,1`, in case of tie, the `sklearn` implementation always returns `0`.
Since choice of labels is arbitrary, this indicates a bias in prediction. 
We ran a simple test by flipping the labels, and saw that the model's predicted accuracy changes quite a bit for the even neighbors case. (code below).

We believe that this *bias* is because of the use of `stats.mode` to find the prediction (which returns the first label sorted low to high).

We also suggest an alternative approach (credits to Pavlos Protopapas, Hayden Joy) in which a label is randomly selected in case of a tie.

```python3
    def _vote(self,labels):
        mean_vote = np.mean(labels)
        
        # if there is a tie, flip a coin
        if mean_vote == 0.5:
            mean_vote += np.random.randn(1)
        return 1*( mean_vote  > 0.5)   
```

Here's a sample plot comparing the two implementations.

![](https://i.ibb.co/ZBD8J9H/results.png)

### Steps/Code to Reproduce

Link to [colab notebook](https://colab.research.google.com/drive/1JCGNVPlrxyKlRHTQsJdAOHragen2JOeV?usp=sharing)

### Expected Results

Similar predictions if labels are reversed.

### Actual Results

Different results when labels are reversed.

Plot comparison.

![](https://i.ibb.co/ZBD8J9H/results.png)

### Versions

```
System:
    python: 3.7.11 (default, Jul  3 2021, 18:01:19)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
       pip: 21.1.3
setuptools: 57.4.0
   sklearn: 0.22.2.post1
     numpy: 1.19.5
     scipy: 1.4.1
    Cython: 0.29.24
    pandas: 1.1.5
matplotlib: 3.2.2
    joblib: 1.0.1

Built with OpenMP: True
```
user: 4722794
labels: Low Priority: Low priority issues and pull requests
issue_21002: state: open
title: Is _generate_sample_indices working correctly ?
body: ### Describe the bug

I fit two random forests (with just one tree each) with identical parameters, but for the second one I disable the row subsampling by setting `bootstrap=False` and instead pass the inbag data generated with _generate_sample_indices.

I believe that the only stochastic elements of a random forest are the row and column subsampling. I disable the former by setting `bootstrap=False` and the latter as well by choosing `max_features = p`.


### Steps/Code to Reproduce

```python
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble._forest import _generate_sample_indices
from sklearn.datasets import load_boston
import matplotlib.pyplot as plt

boston = load_boston()
xtrain, ytrain = boston.data, boston.target
n_samples, p = xtrain.shape

rf = RandomForestRegressor(n_estimators=1, 
    min_samples_leaf =10,max_features = p)
rf.fit(xtrain, ytrain)

tree=rf.estimators_[0]
sampled_indices = _generate_sample_indices(tree.random_state, n_samples, n_samples)

rf0 = RandomForestRegressor(n_estimators=1, 
    min_samples_leaf =10,max_features = p, bootstrap=False)
rf0.fit(xtrain[sampled_indices,:], ytrain[sampled_indices])

p  =  rf.predict(xtrain)
p0 = rf0.predict(xtrain)

plt.scatter(p,p0)
```

### Expected Results

I would expect the two forests (consisting of a single tree) to be identical ?


### Actual Results

The predictions are quite different as you can see in the output

### Versions

System:
    python: 3.8.3 (default, Jul  2 2020, 11:26:31)  [Clang 10.0.0 ]
executable: /opt/anaconda3/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 20.1.1
   setuptools: 49.2.0.post20200714
      sklearn: 0.23.1
        numpy: 1.19.5
        scipy: 1.5.0
       Cython: 0.29.21
       pandas: 1.0.5
   matplotlib: 3.2.2
       joblib: 0.16.0
threadpoolctl: 2.1.0

Built with OpenMP: True

user: markusloecher
issue_21000: state: open
title: Additional metrics in `sklearn.metrics.classification_report`
body: ### Describe the workflow you want to enable

Metrics are **Extremely Important** for benchmarking models' performance. But in **scikit-learn**, it is not easy to extract these metrics from a **multiclass classification** model. In the past, I had to combine different pieces of functions within the `sklearn.metrics` module in order to derive my own. But it is counterproductive, so I end up staying away from **scikit-learn**, and using other libraries such as [pycm](https://www.pycm.ir), [mlxtend](http://rasbt.github.io/mlxtend/), [yellowbrick](https://www.scikit-yb.org/en/latest/), ... etc to get the job done. It is not perfect and still requires some customization, but they are more complete, and much easier to use than **scikit-learn**. So it would be great if **scikit-learn** can improve a little more in this area, so that we can focus more on modeling than customizing the code.



### Describe your proposed solution

`metrics.classification_report` is a good start. The current `metrics.classification_report` return
- Precision (Positive Predictive Value)
- Recall (True Positive Rate)
- F1
- Support

It is great, but far from complete. The following 2 are very important as well
- Negative Predictive Value (NPV)
- Specificity (True Negative Rate, or TNR)

Once we can cover these 4 metrics for the multiclass classification
- Precision (Positive Predictive Value) --> requires (TP, FP)
- Recall (True Positive Rate) --> requires (TP, FN)
- Negative Predictive Value (NPV) --> requires (TN, FN)
- Specificity (True Negative Rate) --> requires (TN, FP)

we can pretty much derive the rest of the metrics, such as
- FNR (False Negative Rate = 1 - TPR)
- FPR (False Positive Rate = 1 - TNR)
- FDR (False Discovery Rate = 1 - PPV)
- FOR (False Omission Rate = 1 - NPV)
- ACC (Accuracy)
- MCC (Mathews Correlation Coefficient)
- Prevalence Threshold
- ...etc

These 2 are quite important as well, but it can be tricky to get for multiclass classification
- AUROC (Area under ROC)
- AUPR (Area under Precision-Recall curve)

The [pycm](https://www.pycm.ir) library is quite comprehensive. Perhaps you may consider integrating some of their goodies into **scikit-learn**.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: Scoodood
issue_20992: state: open
title: Whitespace tokenizing does not work on chinese and japanese
body: ### #FIXME: whitespace tokenizing does not work on chinese and japanese
Implementing NLP whitespace tokenizer for japanese and chinese language can be a little hard.:radioactive: It can't be done like for a eng, ita, fra...etc...

---

### Steps/Code to Reproduce

In :arrow_right:**[/doc/tutorial/text_analytics/data/languages/fetch_data.py](https://github.com/scikit-learn/scikit-learn/blob/40e1f895b172b9941fdcdefffd5a2aa8556ed227/doc/tutorial/text_analytics/data/languages/fetch_data.py#L82)** now we have:
```python

        # split the paragraph into fake smaller paragraphs to make the
        # problem harder e.g. more similar to tweets
        if lang in ('zh', 'ja'):
        # FIXME: whitespace tokenizing does not work on chinese and japanese
            continue
        words = content.split()
        n_groups = len(words) / n_words_per_short_text
        if n_groups < 1:
            continue
        groups = np.array_split(words, n_groups)
                         ...etc...
```
---
### Simple proposal (_for now_):

- Simbol dict. for chinese and japanese lang. will be used for extracting tokens for split the paragraph into fake smaller paragraphs. It's impl. looks like:

```python
        # split the paragraph into fake smaller paragraphs to make the
        # problem harder e.g. more similar to tweets

        n_words_per_short_text_zh_ja = 3

        if lang in ("zh", "ja"):
            words = []
            string_of_words = ''
            for word in content:
                if word in zh_letters :
                    string_of_words += word
                    if len(string_of_words) > n_words_per_short_text_zh_ja:
                        words.append(string_of_words)
                        string_of_words = ''

            n_groups = len(words) / n_words_per_short_text_zh_ja

        else:
            words = content.split()
            n_groups = len(words) / n_words_per_short_text

        if n_groups < 1:
            continue

        groups = np.array_split(words, n_groups)

        for group in groups:
            small_content = " ".join(group)
                    
                                 ...etc...                          
```

- Fror now, we have helper func. to load **zh**, **ja** , letters:

```python

zh_letters = []
def read_zh_letters(filename) -> list:
    with open(filename, encoding="utf8") as file:
        while line := file.readline().strip():
            zh_letters.append(line)
    return zh_letters

read_zh_letters('PATH_TO_zh_ja_letter.txt')
```
---
### Expected And Obtained Result
in "_short_paragraphs/zh_"  is:

- **zh_0000.txt**
`
維基百科 英語或是 维基媒体
`

- **zh_0001.txt**
`
基金会运 营的一个 多语言的
`

- **zh_0002.txt**
`
線上百科 全并以创 建和维护
`

- **zh_0003.txt**
`
作为开放 式协同合 作项目特
`

- **zh_0004.txt**
`
点是自由 容自由编 辑自由版
`

- **zh_0005.txt**
`
权目前是 全球網絡 上最大且
`
` 
             ...etc...
` 
---
### Actual Results

None

---
### Versions

```
System:
    python: 3.8.6 (tags/v3.8.6:db45529, Sep 23 2020, 15:52:53) [MSC v.1927 64 bit (AMD64)]

Python dependencies:
          pip: 21.2.4
   setuptools: 49.2.1
      sklearn: 1.1.dev0
        numpy: 1.21.2
        scipy: 1.7.1
       Cython: None
       pandas: None
   matplotlib: 3.4.3
       joblib: 1.0.1
threadpoolctl: 2.2.0

Built with OpenMP: True

Process finished with exit code 0

```
user: DeanChugall
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_20989: state: open
title: Explain LocalOutlierFactor's Labelling Behaviour to Users
body: ### Describe the issue linked to the documentation

The manner in which LocalOutlierFactor assigns outlier labels may lead to unpredictable outcomes - see #20833 for a specific example and related discussion. 

While I understand why you don't want to fix that behaviour, it seems a good idea to at least acknowledge it in the documentation and advise users to 

* look at the raw LOF scores if results look dodgy
* add their own fix 

### Suggest a potential alternative/fix

A single line in the docs using your preferred wording will do the trick and potentially save users some time and effort.
user: eeghor
issue_20971: state: open
title: incompatible HTML representation for RandomizedSearchCV
body: ### Describe the bug

I don't know if this is intended, but for now the HTML representation of a `RandomizedSearchCV` or `GridSearchCV` shows the estimators and parameters of the initialized pipeline instead of the 'optimized' one found in the search (`best_estimator_`).

### Steps/Code to Reproduce

A simple setup
```
from sklearn.datasets import load_iris
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression, RidgeClassifier
from sklearn.svm import SVC
from sklearn.model_selection import RandomizedSearchCV
from sklearn.utils.fixes import loguniform
iris = load_iris()
pipeline = Pipeline([('clf', SVC())])
search_dists = [
    {'clf': [LogisticRegression(random_state=42)],
     'clf__C': loguniform(1e-2, 1e1),
     'clf__penalty': ['l1', 'l2']
    },
    {'clf': [RidgeClassifier(random_state=42)],
     'clf__alpha': loguniform(1e-2, 1e0)
    },
    {'clf': [SVC(random_state=42)],
     'clf__C': loguniform(1e-2, 1e2), 
     'clf__gamma': loguniform(1e-3, 1e0),
    }]
rand_search_cv = RandomizedSearchCV(pipeline, search_dists, n_jobs=-1, random_state=42)
rand_res = rand_search_cv.fit(iris.data, iris.target)
```
Try to view the HTML representation of the fitted `RandomizedSearchCV`
```
from sklearn import set_config
set_config(display='diagram')
# diplays HTML representation in a jupyter context of the initial `pipeline`
rand_res
```
![image](https://user-images.githubusercontent.com/72174073/132450130-7937084c-e5a4-4c35-a09b-cd73fa982978.png)

### Expected Results

I expected to see the same results as 
```
from sklearn import set_config
set_config(display='diagram')
# diplays HTML representation in a jupyter context of the initial `pipeline`
rand_res.best_estimator_
```
![image](https://user-images.githubusercontent.com/72174073/132458193-f37131d8-b7b7-404c-9ae5-40a25f0f8cbc.png)


### Actual Results

I found the actual presentation irrelevant and doesn't make sense to have initial pipeline as result of visualizing a fitted GridSearch. 

### Versions

System:
    python: 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 16:07:37)  [GCC 9.3.0]
executable: /opt/conda/bin/python
   machine: Linux-5.4.120+-x86_64-with-debian-buster-sid

Python dependencies:
          pip: 21.1.2
   setuptools: 49.6.0.post20210108
      sklearn: 0.24.2
        numpy: 1.19.5
        scipy: 1.6.3
       Cython: 0.29.23
       pandas: 1.2.4
   matplotlib: 3.4.2
       joblib: 1.0.1
threadpoolctl: 2.1.0
user: Abdelgha-4
issue_20967: state: open
title: Warn subclassed UserWarning when unpickling model using different verison of sklearn
body: ### Describe the workflow you want to enable

I want to terminate execution if I'm unpickling a model using a different version of sklearn.

### Describe your proposed solution

Currently it warns with a `UserWarning`:

https://github.com/scikit-learn/scikit-learn/blob/4abd9f2d9e12451fa3e32bd77a97b5c4d8be8b19/sklearn/base.py#L320-L334

If, instead, it used a subclassed UserWarning you could wrap up loading a model in a [catch_warnings](https://docs.python.org/3/library/warnings.html#warnings.catch_warnings) context manager and filter that warning to raise instead, and then exit if it is raised.

### Describe alternatives you've considered, if relevant

You could potentially do this by catching `UserWarning` and checking matching such as `user_warning.message.startswith("Trying to unpickle estimator")` but this is not very robust.

### Additional context

_No response_
user: EdwardBlair
labels: Needs Decision - API: Requires decision regarding API
issue_20964: state: open
title: Show the std of parameters posterior distribution for Bayesian ridge regression
body: ### Describe the issue linked to the documentation

In the documentation for [Bayesian ridge regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.BayesianRidge.html) the standard deviation for the posterior distribution of the target variable is easily accessed by calling `.predict(X, return_std=True)`.

However, there is no comparable method for accessing standard deviations for the posterior distributions of the learned parameters. The means of the parameters posterior distributions are given by `.coef_` , but the corresponding standard deviation is not easily available. 

Am I correct that the standard deviation would be given by the square root of the diagonal of the covariance matrix?
`np.sqrt(clf.sigma_.diagonal())`

### Suggest a potential alternative/fix

Provide an example of accessing the standard deviation of the posterior distribution of the learned parameters.
user: davidsteinar
issue_20954: state: open
title: Add `fit_params` to `IterativeImputer`'s `fit` method
body: ### Describe the workflow you want to enable

Imputing missing values with `IterativeImputer` can be slow if the dataset size is large. If you pass a tree-based model to the wrapper, it is even longer. 

To speed up training, I wanted to use GPU-enabled models with their sklearn-compatible APIs like XGBoost, LightGBM, or CatBoost. 



### Describe your proposed solution

All of these models accept additional parameters in their `fit` method, some of which are very valuable to the training such as `early_stopping_rounds`, `eval_metric`, `eval_set`, or categorical feature support in the case of LGBM or CatBoost. 

CV utilities such as `cross_val_score` accept these params if you pass them to the `fit_params` parameter but  I couldn't find one in `IterativeImputer`'s `fit` method. Is it possible to include this feature to the estimator?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: BexTuychiev
issue_20953: state: open
title: Add random forest row-subsampling without replacement
body: ### Describe the workflow you want to enable

I do appreciate the current option of disabling bootstrapping via the Boolean argument `bootstrap`.
However, currently there is only one alternative: If False, the whole (identical) dataset is used to build each tree.
There is well known research though (starting with the paper by [Strobl et al.](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-8-25) from 2007) that in certain situations subsamples drawn without replacement leads to better performance.
Many well known random forest implementations (such as [ranger](https://cran.r-project.org/web/packages/ranger/index.html) ) offer this sub sampling as an alternative to the bootstrap.

I would greatly appreciate it if sklearn would offer the same.


### Describe your proposed solution

**For both `RandomForestRegressor`  and  `RandomForestClassifier` allow the user to draw subsamples without replacement for each tree instead of the bootstrap.**
The user can choose the fraction of sub samples drawn for each tree (default: 0.632 )
Ideally the functions `_generate_unsampled_indices` and `_generate_sampled_indices` would still work.


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

### API considerations
Currently, random forests have the option `bootstap=True` or `False`. If this new feature of tree-wise row-subsampling without replacement is added, there are several options:
1. Add a new option `with_replacement=True` (default), or `False`, that only takes effect if `bootstrap=True`.
   Disadvantage: The term bootstrap explicitly means sampling with replacement.
2. Add a new option `row_subsampling=True` (default) or `False`, which samples with replacement if `bootstrap=True` and without replacement if `bootstrap=False`.
   Disadvantage: It would change current behaviour for `bootstrap=False`, which currently means no sampling at all.
3. Add new option `sampling="bootstrap"` (default), and allow callable / splitter to be passed. Deprecate option `bootstrap` (proposed in https://github.com/scikit-learn/scikit-learn/issues/20953#issuecomment-923957749).
user: markusloecher
labels: Needs Decision: Requires decision
issue_20952: state: open
title: TransformedTargetClassifier
body: ### Describe the workflow you want to enable

I would like to be able to use a LabelEncoder as a wrapper for a classifier, similarly to what can be achieved with preprocessors on the y value for regressors via the [TransformedTargetRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.compose.TransformedTargetRegressor.html).

### Describe your proposed solution

Add a class TransformedTargetClassifier that accepts both a transformer on y and a classifier.

### Describe alternatives you've considered, if relevant

An alternative would be to use the voting classifier with a single estimator, but that appears to be misusing that class.

### Additional context

I'm proposing this feature because in Auto-sklearn we use the LabelEncoder on a call to fit to have all classifiers we try use a simple, encoded representation. When using the Auto-sklearn classes we can undo the transformations ourselves. However, if the user would like to access an individual model, there's no way we can wrap the LabelEncoder around this models.

CC @eddiebergman
user: mfeurer
issue_20951: state: open
title: Math not rendered in downloaded documentation when offline
body: ### Describe the issue linked to the documentation

I downloaded the zipped HTML documentation for Scikit-learn 0.24.2 from https://scikit-learn.org/dev/versions.html. However, the math in the documentation is not rendered when I am not connected to the internet. This is what I see when I view section 1.1 of the User Guide when offline:

![2021-09-06_15-55-2](https://user-images.githubusercontent.com/11555232/132182340-8725aa8d-3409-4674-871e-e19c9c51f518.png)



The problem is caused by the following line, which will fail to load the MathJax script when offline.

```html
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-chtml.js"></script>
```

### Suggest a potential alternative/fix

The solution is to bundle the MathJax script in the zipped HTML docs.
user: cwfoo
issue_20943: state: open
title: Add more D2 scores
body: ### Describe the workflow you want to enable

I'd like to evaluate models by "percentage of reduction in metric X", where X is:
- LogLoss
- absolute error
- pinball loss

As `d2_tweedie_score` #17036, these are generalizations of the `r2_score`.

### Describe your proposed solution

Add
- `d2_log_loss_score`
- `d2_absolute_error_score`
- `d2_pinball_loss_score`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

See https://github.com/scikit-learn/scikit-learn/pull/17036#issuecomment-903939035.
user: lorentzenchr
issue_20923: state: open
title: TfidfTransformer does not support large sparse matrices on 32bit platforms
body: `TfidTransformer` does not support large sparse matrices (indices as int64) because `_document_frequency` uses `np.bincount`. `np.bincount` works on arrays with dtype `NPY_INTP` which is int32 on 32bit platforms.

REF: https://github.com/scikit-learn/scikit-learn/pull/20919
user: thomasjpfan
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_20920: state: open
title: get_params method in cross-validation splitters
body: ### Describe the workflow you want to enable

I would like to be able to recreate the cross-validataion splitters from parameters.

Presently, the cross-validation splitters do not have `get_params` or `set_params` options. I want to make different steps in my process stateless, and to do that I need to be able to retrieve the parameters used to create an object (and pass them out), then recreate the object (obviously in an unfitted state, but that doesn't matter for cv splitters) when the parameters are passed back in.


### Describe your proposed solution

`get_params` and `set_params` are already implemented in `BaseEstimator`. There are two possible options:

1. Copy the implementation into `BaseCrossValidator`
   This is faster to implement, but it isn't very DRY, and it isn't very maintainable. This could be done as a quick patch, but doing so might make problems for someone else later.
2. Pull the common API stuff backward to some kind of BaseSKObject class
  This is more maintainable, cleaner, and overall a better solution, but it will take thought, architectural consideration, and intentionality. This is not a quick patch.

### Describe alternatives you've considered, if relevant

I can (and probably will) store the values I used to create the object in the first place, and pass them out when I build my parameter string. I don't think this will be completed right away (if it's even desired by the community at large), and the show must go on.

But having `get_params` and `set_params` common across all sklearn objects feels like a necessary step toward a more unified codebase. And it would make my life a lot easier!

### Additional context

_No response_
user: swight-prc
issue_20917: state: open
title: Add an option to plot the iso-fbeta lines in PrecisionRecallDisplay
body: ### Describe the workflow you want to enable

We recently added the `PrecisionRecallDisplay`. This display will plot the precision-recall curve.
It can be improved by adding an argument in the `plot` method to enable plotting the fbeta isoline similarly done in the following examples:

https://scikit-learn.org/dev/auto_examples/model_selection/plot_precision_recall.html#plot-precision-recall-curve-for-each-class-and-iso-f1-curves

It would remove the boilerplate and could be useful in some cases.

### Describe your proposed solution

We will need to add some arguments to the `plot` method.

I am unsure about the API thought. We might add 2 parameters: one to control whether or not to plot the isoline and another one to control the beta of the F-beta score.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: glemaitre
issue_20898: state: open
title: Allow `Pipeline` to whether pass or not `y` to some transformer
body: ### Describe the workflow you want to enable

Some transformers could be used in both a supervised and unsupervised setting. Currently, `y` is always passed at `fit` and it imposes that such transformers can only be used in a supervised way.

Such transformers could be:

- `PLSRegression`: https://github.com/scikit-learn/scikit-learn/issues/20783
- `UMAP`: https://github.com/scikit-learn/scikit-learn/issues/20815

### Describe your proposed solution

We might be able to change the `Pipeline` implementation.

### Describe alternatives you've considered, if relevant

We could also delegate this option to the constructor the transformer that should specify if it should work in supervised or unsupervised mode. Something similar to `novelty=True/False` for switching from novelty detection vs. outlier detection for some estimator.

### Additional context

To give a succinct example:

```python
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification
import umap

X, y = make_classification(random_state=0)

pipe = Pipeline([
    ('scaler', StandardScaler()), ('umap', umap.UMAP()), ('lr', LogisticRegression()
])
pipe.fit(X, y)
```
user: glemaitre
issue_20890: state: open
title: Allow users to pass their own grid to partial dependence
body: ### Describe the workflow you want to enable

The partial dependence function does it's own grid calculation that assumes all the data is numeric via the `np.unique` method call. This is problematic for pipelines where the input data is transformed into numeric values along the way. 

For example, this is an example trying to compute partial dependence on a categorical column with missing values. It errors out because `np.unique` does not work with categorical columns with missing values. 

```python
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import partial_dependence
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
import pytest

pl = Pipeline([("Imputer", SimpleImputer(strategy="most_frequent")),
               ("OHE", OneHotEncoder()),
               ("RFC", RandomForestClassifier())])

X = pd.DataFrame({"a": [1, 2, 3, 4], "b": ["a", "b", "a", "b"]})
y = pd.Series([0, 1, 0, 1])
pl.fit(X, y)

X_holdout = pd.DataFrame({"a": [1, 2, 3, 4], "b": ["a", "b", "a", None]})

with pytest.raises(TypeError, match="'<' not supported between instances of 'NoneType' and 'str'"):
    partial_dependence(pl, X_holdout, features=1, grid_resolution=3)
```

I think this goes beyond just `np.unique`. I could have a pipeline that transforms a datetime into a numeric value. There are multiple ways of computing a grid for datetimes, like transforming into unix epoch time and computing quantiles or picking times every x timesteps apart. 

### Describe your proposed solution

Letting users pass in their own grid and values parameters.

```python
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import partial_dependence
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
import pytest

pl = Pipeline([("Imputer", SimpleImputer(strategy="most_frequent")),
               ("OHE", OneHotEncoder()),
               ("RFC", RandomForestClassifier())])

X = pd.DataFrame({"a": [1, 2, 3, 4], "b": ["a", "b", "a", "b"]})
y = pd.Series([0, 1, 0, 1])
pl.fit(X, y)

X_holdout = pd.DataFrame({"a": [1, 2, 3, 4], "b": ["a", "b", "a", None]})
partial_dependence(pl, X_holdout, features=1, grid_resolution=3, grid=np.array([['a'], ['b']]), values=[['a', 'b']])
```

### Describe alternatives you've considered, if relevant

We could make the existing implementation more robust to categorical values by using a different function than `np.unique`.

### Additional context

If this is a welcome change but low-priority, I'd be happy to help!
user: freddyaboulton
issue_20859: state: open
title: Add PDF download of documentation
body: ### Describe the issue linked to the documentation

In the list of documentation downloads (https://scikit-learn.org/dev/versions.html), the documentation for the latest scikit-learn version (0.24.2) is not available as a PDF download. The documentation for prior versions of scikit-learn are available in PDF format. See #17051, #17564 and #19208.

![2021-08-27_17-57](https://user-images.githubusercontent.com/11555232/131109754-81aaf691-29a4-4747-886d-d709405fd291.png)


### Suggest a potential alternative/fix

Please add a PDF of the documentation for scikit-learn 0.24.2.
user: cwfoo
issue_20856: state: open
title: [WIP] SequentialFeatureSelector stores the intermediate scores and the history of the support masks
body: #### What does this implement/fix? Explain your changes.
Currently `SequentialFeatureSelector` returns only the final mask, but discards the results of all the intermediate computations. 
Those results are often of interest, so it would be worthwhile to store them and allow users to access them.
This PR stores the intermediate scores (results of various tested model evaluations), and the history of the support masks.

I would like to ask for a review of the idea and the current implementation!
If I get a green light, I will complete the task list below before the final review.

The naming of the variables is provisional, I was trying to adhere to the standard I've found in this file, but I'm very open to suggestions!

Suggested reviewers (is it a good habit to do this?): @NicolasHug, @rasbt, @thomasjpfan, @glemaitre

#### Task list 
- [ ]  Review of the concept & current implementation
- [ ]  Add public functions to access the variables created
- [ ]  Add docs
- [ ]  Add this feature to one of the user guide examples
- [ ]  Add `scoring` argument to one of the user guide examples (oddly enough I had a difficult time finding this argument, would be useful to include it in an example!)
user: talesa
issue_20852: state: open
title: Rank about GridSearchCV
body: ### Describe the bug

row 934-936 in `_search.py`:
```python
            if rank:
                results["rank_%s" % key_name] = np.asarray(
                    rankdata(-array_means, method='min'), dtype=np.int32)
```
maybe it is more correct to rank with both of `array_means` and `array_stds`, bigger mean and smaller std means better performence, if the mean is equal, may we need to continue to rank by std

### Steps/Code to Reproduce

row 934-936 in `_search.py`:
```python
            if rank:
                results["rank_%s" % key_name] = np.asarray(
                    rankdata(-array_means, method='min'), dtype=np.int32)
```

### Expected Results

rank with both of `array_means` and `array_stds`

### Actual Results

rank by  `array_means` 

### Versions

0.24.2
user: BinchaoPeng
labels: Needs Decision - Close: Requires decision for closing
issue_20819: state: open
title: Adding Oblique Trees (Forest-RC) to the Cythonized Tree Module
body: ### Describe the workflow you want to enable

I am part of the @neurodata team. We are proposing to add a cythonized module that allows for building oblique trees.

Oblique trees, otherwise known as `Forest-RC`, originally proposed by, Breiman 2001 [1], would enable a more flexible class of decision trees. Breiman called a version of these Forest-RC, as compared to what most people call random forest, which he called Forest-RI. He noted:

"Forest-RC does exceptionally well on the synthetic data sets. Overall, it compares more favorably to Adaboost than Forest-RI."

In other words, in Breiman's paper introduction random forest over 20 years ago, with over 75,000 citations, he presented 2 algorithms, and argued that the one which does sparse linear combinations of features worked better than the naive one which most people have used since. More recently, several authors have corroborated these original results [2-6]. Oblique trees in general open up a wide class of flexibility for trees to take advantage of structure in data. 

[1]: Breiman, L. Random Forests. Machine Learning 45, 5–32 (2001). https://doi.org/10.1023/A:1010933404324
[2]: T. M. Tomita, J. Browne, C. Shen, J. Chung, J. L. Patsolic, B. Falk, J. Yim, C. E. Priebe, R. Burns, M. Maggioni, and J. T. Vogelstein. [Sparse Projection Oblique Randomer Forests](https://jmlr.org/papers/v21/18-664.html). Journal of Machine Learning Research, 2020.
[3]: Tyler M Tomita, Mauro Maggioni, and Joshua T Vogelstein. Roflmao: Robust oblique forests with linear matrix operations. In Proceedings of the 2017 SIAM International Conference on Data Mining, pages 498–506. Society for Industrial and Applied Mathematics, 2017.
[4]: Rainforth, Tom & Wood, Frank. (2015). Canonical Correlation Forests. 
[5]: Perry, R., Li, A., Huynh, C., Tomita, T.M., Mehta, R., Arroyo, J., Patsolic, J., Falk, B., & Vogelstein, J. (2019). Manifold Oblique Random Forests: Towards Closing the Gap on Convolutional Deep Networks.
[6]: Menze, Bjoern & Kelm, Bernd & Splitthoff, Daniel & Köthe, Ullrich & Hamprecht, Fred. (2011). On Oblique Random Forests. 453-469. 10.1007/978-3-642-23783-6_29. 

### Describe your proposed solution

The implementation of Forest-RC (oblique forests) was ported into a scikit-learn fork (https://github.com/neurodata/scikit-learn/pull/11/). The Forest-RC Cython implementation currently builds cleanly on top of the existing cython code for the regular Decision Trees and thus does not affect the runtime, or performance on Random Forests at all. Moreover, it improves upon RF in certain cc_18 benchmarks (see figure below). 

The `.pxd` files can also be included in each scikit-learn release, so that other sklearn-compatible forest modules can build upon our implementation of the Oblique tree.

### Relevant Benchmarks

To ease scikit-learn developer concerns on incorporating such a large piece of code change, we can provide the following benchmarks and tests on Cythonized Forest-RC vs Cythonized Forest-RI. These benchmarks were ran on the OpenML CC-18 dataset. Max features were set to n_features, so runtime was very high for both RF and oblique forests in e.g. Fashion-MNIST.

1. **benchmark tests on cc_18 vs Random Forests** (see two figures below) Forest-RC is significantly better then RF on this benchmark suite. A paired one-sided Wilcoxon signed test was ran. Forest-RC was significantly better then RF with a PValue ~ 1e-20.

![image](https://user-images.githubusercontent.com/3460267/132111028-689a64cc-15c3-4177-a92e-a0503d873d33.png)
![image](https://user-images.githubusercontent.com/3460267/130329139-b27484f3-365b-4e03-a977-2b0e35f66c7e.png)

2. **overall runtime (training times on cc_18) vs Random Forests**

![image](https://user-images.githubusercontent.com/3460267/132111128-c2ed2ec6-74f9-4766-b147-26df53069e59.png)

3. **Runtime benchmarking dependency on number of features and number of samples:** In addition, we performed a benchmarking the runtime performance of oblique trees/forests using the code provided in `benchmarks/bench_tree.py`. The notebook is shown in this gist: https://gist.github.com/adam2392/a652aa1c88ba94f0d6aab96ccd3ade24

4. **visualizing the intuitive differences between Oblique and Axis-aligned trees:** Per a discussion with @thomasjpfan , he wanted to see how we can educate an incoming user when/how oblique trees are different. We can extend the [existing example in sklearn](https://scikit-learn.org/stable/auto_examples/tree/plot_iris_dtc.html#sphx-glr-auto-examples-tree-plot-iris-dtc-py). The following notebook shows the decision surface of an oblique tree vs axis-aligned tree: https://gist.github.com/adam2392/78519091104cecfeb8ff796eac7e8115
 

### Additional context

cc: @jovo 

Some action items are:

1. Unit testing (I will probably need some direction as to where tests can be added as say parametrization, and what sort of tests the maintainers would like to see)
2. Make the official PR so developers/maintainers can track progress

user: adam2392
issue_20813: state: open
title: RFC Consistency for meta infos in files (license, encoding, authors, ..)
body: There are several pieces of information that some files contain, others do not, examples are:

- `# -*- coding: utf-8 -*-` at the beginning of a file
- License information, eg `# License: BSD 3 clause`
- List of (some main) authors for that file, `# Authors:`

For the following reasons, I'd like to remove the encoding info, and have a consistent policy for the other 2 points:
- As a (newish) contributor, it's hard to figure out what is expected.
- Having those infos consistent in all files might help a tiny little bit to generate trust in the production-readiness of scikit-learn.

I haven't figured out a convincing policy for `authors`, maybe leave it as is.

Note that changes of those lines of code shouldn't produce merge conflicts, cross referencing #11336.
user: lorentzenchr
issue_20804: state: open
title: Revisting the tags interface
body: With https://github.com/scikit-learn/scikit-learn/pull/20657 (`__sklearn_is_fitted__`) merged I am starting to rethink the tags implementation how we walk the MRO. I think tags would be much easier to use if it was a protocol like `__sklearn_tags__`. This way all mixins and subclasses must call `super().__sklearn_tags__()` and update it themselves. This makes it so that third party developers do not need to import `BaseEstimator` or the whole tags machinery to take advantage of tags.

I recall this was discussed in https://github.com/scikit-learn/scikit-learn/pull/8022 but I do not know why it was rejected.


BTW If we had a simpler and easier to extend tags interface, I use a `"is_fitted"` tag into it with default value: `"implicit"`.

user: thomasjpfan
labels: Hard: Hard level of difficulty
Needs Decision - API: Requires decision regarding API
issue_20801: state: open
title: Support for 2d sample weights for MultiOutputEstimators
body: ### Describe the workflow you want to enable
Consider a multioutput estimation problem where there are missing values in the outputs.

Imagine that `y` is
```python
array([[ 1., -1., nan],
       [-1., nan,  1.],
       [-1.,  1.,  1.]])
```
Then the sample weights would be
```
array([[1., 1., 0.],
       [1., 0., 1.],
       [1., 1., 1.]])
```

### Describe your proposed solution

Change the end of `sklearn.multioutput._MultiOutputEstimator.fit` from
```python
        self.estimators_ = Parallel(n_jobs=self.n_jobs)(
            delayed(_fit_estimator)(
                self.estimator, X, y[:, i], sample_weight,
                **fit_params_validated)
            for i in range(y.shape[1]))
        return self
```
to something like
```python
        slice_sample_weight = sample_weight is not None and len(sample_weight.shape) == 2
        
        self.estimators_ = Parallel(n_jobs=self.n_jobs)(
            delayed(_fit_estimator)(
                self.estimator, X, y[:, i], sample_weight[:, i] if slice_sample_weight else sample_weight,
                **fit_params_validated)
            for i in range(y.shape[1]))
        return self
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
user: krey
issue_20775: state: open
title: HalvingGridSearchCV number of classes
body: #### Describe the bug

Errors:
You may have to run the code several times to see these errors. I think most of them is to do with the way the initial number of samples is selected; If the initial number of samples contains the same target then the fit function fails. ~I also seem to be running into problems with no samples being passed?~

ValueError: The number of classes has to be greater than one; got 1 class
~ValueError: Found array with 0 sample(s) (shape=(0, 22)) while a minimum of 1 is required.~
ValueError: zero-size array to reduction operation maximum which has no identity


oblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.
ERROR: The process "8404" not found.
ERROR: The process "9492" not found.
ERROR: The process "28656" not found.
ERROR: The process "29236" not found.
ERROR: The process "29412" not found.
ERROR: The process "9004" not found.
ERROR: The process "28564" not found.
ERROR: The process "12072" not found.
ERROR: The process with PID 3748 (child process of PID 11860) could not be terminated.
Reason: Access is denied.
ERROR: The process "4372" not found.
ERROR: The process "2436" not found.
ERROR: The process with PID 28212 (child process of PID 13280) could not be terminated.
Reason: Access is denied.
ERROR: The process "25468" not found.
ERROR: The process with PID 27876 (child process of PID 9884) could not be terminated.
Reason: Access is denied.
ERROR: The process with PID 27912 (child process of PID 19872) could not be terminated.
Reason: Access is denied.

#### Steps/Code to Reproduce

Example:
```python
import numpy as np
from sklearn.decomposition import KernelPCA
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import HalvingGridSearchCV
from sklearn.pipeline import Pipeline

from sklearn.svm import SVC
from sklearn.model_selection import ShuffleSplit
from sklearn.metrics import make_scorer, f1_score

data_input = np.random.randn(4000,22)
data_output = np.round(np.random.random((4000,))).astype((int))

search_points = 10
gamma = np.logspace(-9, 1, search_points)
coef = np.linspace(-1, 1, search_points)
points = np.round(np.linspace(1, len(data_input[0, :]), search_points))
FE_algorithm = KernelPCA(kernel='rbf')
FE_grid = {'FE__n_components': points.astype(int), 'FE__gamma': gamma,'FE__coef0': coef,}

regularisation = np.logspace(0,1,search_points)
Class_algorithm = SVC(kernel='linear')
CA_grid = {'CA__C': regularisation,}

pipe = Pipeline(steps=[('FE', FE_algorithm), ('CA', Class_algorithm)])
param_grid = {**FE_grid, **CA_grid} # concat dicts

scorer = make_scorer(f1_score, average='weighted')
search = HalvingGridSearchCV(pipe, param_grid, cv=ShuffleSplit(test_size=0.20, n_splits=1, random_state=0),
                                      n_jobs=-1, verbose=3, scoring=scorer)

search.fit(data_input, data_output)
```

#### Versions

Windows-10-10.0.19041-SP0
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
NumPy 1.19.5
SciPy 1.6.3
Scikit-Learn 0.24.1




user: Joel-H-dot
issue_20762: state: open
title: FEA Allow negative integers in categorical_features for HGBT
body: #### Describe the workflow you want to enable
I'd like to pass negative integers to indicate categorical feature columns from the end.
```
X, y = ...

ct = make_column_transformer(
    (SomeTransformer(), make_column_selector(dtype_exclude='category')),
    (OrdinalEncoder(), make_column_selector(dtype_include='category')),
    )

hgbt_pipe = make_pipeline(
    ct,
    HistGradientBoostingRegressor(categorical_features=[-1, -2, -3])
)
```

One could also consider to allow `slice`.
user: lorentzenchr
issue_20760: state: open
title: FEA SplineTransformer difference penalty
body: #### Describe the workflow you want to enable
When using `SplineTransformer` as part of a larger feature preprocessing pipeline that ends in a linear model, e.g. `Ridge`, I'd like the splines to get a penalty on the (first) differences of the coefficients instead of the usual L2 penalty:
```python
alpha * sum(squared_norm(np.diff(coef))) 
```
instead of
```python
alpha * sum(squared_norm(coef))
```

The reason is that such a penalty penalizes the curvature (integral over square of second derivatives of splines) and therefore has better interpolation and smoothness properties.
The B-splines would then be called P-splines, see Eilers & Marx "Flexible smoothing with B-splines and penalties" https://doi.org/10.1214/ss/1038425655
Properties (see Chapter 5):
- P-splines show no boundary effects, as many types of kernel smoothers do.
- P-splines can fit polynomial data exactly (with polynomial degree=k, a P-spline with k+1 difference penalty or higher)
- P-splines can conserve moments of the data. For a GLM with canonical link with P-splines of degree k+1 and a penalty of order k+1, or higher, it holds that `sum_i x_i^k y_true_i = sum_i x_i^k y_predict_i`.
  => 1st order P-splines conserve zero, first and 2nd momentum on training data.

#### Describe your proposed solution
I see 2 solutions:

1. Somehow (SLEP006 ?) pass the information that those columns are B-splines and should get a different penalty to the final linear regressor.
2. Add an additional option to `SplineTransformer`, like `diff=True`, that triggers to return `np.diff(X)` instead of `X`. The usual L2 penalty of, e.g. `Ridge`, is then equivalent to L2 of the difference of coefficients of the original `X`.
3. Further ideas?
user: lorentzenchr
issue_20717: state: open
title: Support reusing of joblib thread pools in parallel routines
body: #### Describe the workflow you want to enable

I'm using `pairwise_distances` for fairly large matrices and, at the same time, long vectors. It's worth starting up the parallel threads for the matrices but not for the vectors. It would be great, however, if the thread pool could be reused to benefit for parallelism for the smaller computations as well.

#### Describe your proposed solution

`pairwise_distances` calls  `_parallel_pairwise` internally which creates a new thread pool (by instantiating `Parallel`) every time. Being able to pass the joblib parallel context as an argument would enable reusing the thread pool.

https://github.com/scikit-learn/scikit-learn/blob/3ae7c7615343bbd36acece57825d8b0d70fd9da4/sklearn/metrics/pairwise.py#L1416

#### Describe alternatives you've considered, if relevant

I can run the smaller calculations on a single thread now, it's faster than starting up the thread pool.

#### Additional context

-
user: dobos
issue_20696: state: open
title: Meta-estimators and delegating validation
body: Should meta-estimators set `n_features_in_` if the sub-estimator does not define it? For example, if the first step of a Pipeline does not have `n_features_in_`, should `Pipeline` try to set it and handle the validation?

I think it's strange that meta-estimators do not follow [SLEP010](https://scikit-learn-enhancement-proposals.readthedocs.io/en/latest/slep010/proposal.html) if the sub-estimator does not.
user: thomasjpfan
issue_20672: state: open
title: BallTree and KDTree query radius out of memory (optimisation or bug)
body: #### Describe the bug

The BallTree and KDTree have a method query_radius that returns all the points near a certain radius. 
I have noticed that when you query via iteration point by point there is no issue with memory allocation but when you perform on the entire array it saturates the memory. I am not sure whether this is a memory leak or requires some form of optimization.

#### Steps/Code to Reproduce

Example:
```python
import gc

import sklearn
from sklearn.neighbors import NearestNeighbors
import numpy as np
from memory_profiler import memory_usage
import psutil

def get_memory_usage():
    return memory_usage(timeout=0.2)[0]

vmem = psutil.virtual_memory()
print('sklearn:', sklearn.__version__)
print('Initial memory usage: %.2f MB' % get_memory_usage())
print('RAM available: %.2f GB' % (vmem.available/1e9))

N = 60000
X = np.random.rand(N, 2)  # Random data

print('Allocated X: %.2f MB' % get_memory_usage())
print('X.nbytes == %.2f MB' % (X.nbytes / 1e6))
print('X min %.2f' % X.min())
print('X max %.2f' % X.max())
```

Now let's try to find all neighbours:

```
# query via all elements crashes memory
ind = tree.query_radius(X, r=0.5) 
```

That will exhaust all your memory, increase or decrease N for testing.

#### Expected Results

If I query via iteration it all works well:

```
# query by iteration all elements
all = []
for i in range(X.shape[0]):
  ind = tree.query_radius(X[i:i+1], r=0.5) 
  found = ind[0].shape[0]
  all.append(found)
  assert found>0
print('Allocated with tree: %.2f MB' % get_memory_usage())
```
About 21 seconds on Colab and around 10 MB additional memory.

#### Actual Results

Out of memory on a 13 GB RM colab setup. Please change N parameter in case you have a larger setup.

#### Versions

System:
    python: 3.7.11 (default, Jul  3 2021, 18:01:19)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
       pip: 21.1.3
setuptools: 57.2.0
   sklearn: 0.22.2.post1
     numpy: 1.19.5
     scipy: 1.4.1
    Cython: 0.29.23
    pandas: 1.1.5
matplotlib: 3.2.2
    joblib: 1.0.1

Built with OpenMP: True

user: priamai
issue_20669: state: open
title: Support `RandomState._bit_generator` as input in `check_random_state` for users with numpy version >= 1.17.0
body: #### Describe the workflow you want to enable
Since numpy version 1.17.0, `np.random.RandomState` can accept the `._bit_generator` attribute as input in the constructor. This can be a plus for those who use `np.random.Generator` in their code and want to use the same bitgenerator with sklearn's estimators. Currently this is not possible, see:

```python
from sklearn.datasets import make_classification
from sklearn.manifold import TSNE

X, y = make_classification(n_samples=150, n_features=5, n_informative=5,
                           n_redundant=0, n_repeated=0, n_classes=3,
                           n_clusters_per_class=1,
                           weights=[0.01, 0.05, 0.94],
                           class_sep=0.8, random_state=0)

rng = np.random.default_rng(12345)
tsne = TSNE()
# some piece of code here
# then later we use our own rng to set the seed of `tsne`
# notice `_bit_generator` used here, which is compatible with RandomState
tsne.set_params(random_state=rng._bit_generator)
tsne.fit_transform(X, y)
```

this leads to the error:
```
  File "/home/python3.6/site-packages/sklearn/manifold/_t_sne.py", line 932, in fit_transform
    embedding = self._fit(X)
  File "/home/python3.6/site-packages/sklearn/manifold/_t_sne.py", line 728, in _fit
    random_state = check_random_state(self.random_state)
  File "/home/python3.6/site-packages/sklearn/utils/validation.py", line 944, in check_random_state
    ' instance' % seed)
ValueError: <numpy.random.pcg64.PCG64 object at 0x7ffa3ab471b8> cannot be used to seed a numpy.random.RandomState instance
```

#### Describe your proposed solution
I propose we add a conditional in `check_random_state` that supports an instance of `BitGenerator`, see:
https://github.com/scikit-learn/scikit-learn/blob/2beed55847ee70d363bdbfe14ee4401438fba057/sklearn/utils/validation.py#L926-L944
something like
```python
supported_bitgenerators = {'PCG64', 'SFC64', 'Philox', ...}

def check_random_state(seed):
	...
	if seed.__class__.__name__ in supported_bitgenerators:
		return np.random.RandomState(seed) # should work if numpy>=1.17.0
	...
```
#### Describe alternatives you've considered, if relevant
I know there is an issue regarding supporting the new numpy Generator interface but I feel this is slightly different since it does not attempt to replace `RandomState`.


user: zoj613
issue_20666: state: open
title: Threads awaiting for GIL in Forest estimators
body: ### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/20651

In forest algorithms, the preferred parallelization backend is `threading`. However, it looks that it is not anymore the most appropriate backend. As discussed [here](https://github.com/scikit-learn/scikit-learn/discussions/20651#discussioncomment-1124431), it might be that the GIL is not explicitly released in some part of the code locking the execution of the thread.

We need to investigate more to solve this issue. 
user: glemaitre
issue_20640: state: open
title: CI issue with OpenMP
body: We recently have a failure in 2 of the builds due to OpenMP.
The relevant messages are:

```
gcc -pthread -B /usr/share/miniconda/envs/testvenv/compiler_compat objects/test_program.o -o test_program -fopenmp
/usr/share/miniconda/envs/testvenv/compiler_compat/ld: warning: libdl.so.2, needed by /usr/lib/gcc/x86_64-linux-gnu/9/libgomp.so, not found (try using -rpath or -rpath-link)
/usr/share/miniconda/envs/testvenv/compiler_compat/ld: /usr/lib/gcc/x86_64-linux-gnu/9/libgomp.so: undefined reference to `dlopen@GLIBC_2.2.5'
/usr/share/miniconda/envs/testvenv/compiler_compat/ld: /usr/lib/gcc/x86_64-linux-gnu/9/libgomp.so: undefined reference to `dlerror@GLIBC_2.2.5'
/usr/share/miniconda/envs/testvenv/compiler_compat/ld: /usr/lib/gcc/x86_64-linux-gnu/9/libgomp.so: undefined reference to `dlclose@GLIBC_2.2.5'
/usr/share/miniconda/envs/testvenv/compiler_compat/ld: /usr/lib/gcc/x86_64-linux-gnu/9/libgomp.so: undefined reference to `dlsym@GLIBC_2.2.5'
```
user: glemaitre
issue_20634: state: open
title: RFC make shape of arrays consistent
body: Recently, we got some issues regarding the shape inconsistencies:

- of `coef_` in linear model: https://github.com/scikit-learn/scikit-learn/pull/20604
- of `y_pred` when providing a single target of shape `(n_samples, 1)`: https://github.com/scikit-learn/scikit-learn/pull/20603

While it is pretty easy to make those arrays consistent across estimators, one could argue that the current user code is functioning.

This RFC is dedicated to know if we should solve this issue (see the failure in the different PRs) and if yes, shall we use a deprecation cycle to not break the code of our users from today to tomorrow?
user: glemaitre
issue_20630: state: open
title: MLflow cannot patched estimator methods decorated with "property", we should decorate them with "available_if" instead
body: #### Describe the workflow you want to enable

There're some special methods in sklearn estimators, e.g. `SVC.predict_proba`, it is defined like:
```
class SVCBase:
    ...
    @property
    def predict_proba(self):
        self._check_proba()  # raise error if predict_proba not available
        return self._predict_proba
```
When we call `svc_model.predict_proba(X)`, the behavior is, if available, run delegated `_predict_proba` , otherwise `hasattr(svc_model, 'predict_proba')` will return `False`.

`SVC.predict_proba` looks like property, but it behaves like a normal function, it is a trick to add available-checking logic.

But, such trick block `mlflow` to patch it ([MLflow](https://mlflow.org/docs/latest/index.html) patch some methods of estimators (e.g. patch fit/predict/transform etc.) in order to do autologging)

The full list contains these methods which has similar issue:
```
FeatureAgglomeration.fit_predict
LocalOutlierFactor.fit_predict
LocalOutlierFactor.predict
MultiOutputClassifier.predict_proba
NuSVC.predict_proba
SGDClassifier.predict_proba
SVC.predict_proba
VotingClassifier.predict_proba
NuSVC.predict_log_proba
SGDClassifier.predict_log_proba
SVC.predict_log_proba
```

#### Describe your proposed solution

decorate them with "available_if" instead, so they will be a normal method which mlflow can patch them.
`Pipeline.transform` is an example which previously decorated with "property" but now replaced with "available_if".
https://github.com/scikit-learn/scikit-learn/blob/030049879fc1d45fe7832172a8f259d1e7afffdf/sklearn/pipeline.py#L559

#### Describe alternatives you've considered, if relevant

#### Additional context

[MLflow](https://mlflow.org/docs/latest/index.html) patch some methods of estimators (e.g. patch fit/predict/transform etc.) in order to do autologging. MLflow is a very popular project so let's make sklearn support mlflow better.

user: WeichenXu123
issue_20608: state: open
title: TST split API checks from other checks
body: related to #16241
related to #13969
related to #17361

Working on more modular checks.
user: glemaitre
issue_20604: state: open
title: TST add common tests for linear models for shape consistency of coef
body: Related to #19693

Add common test to check the shape of `coef` in the linear models
user: glemaitre
issue_20603: state: open
title: TST common test for predictions shape consistency with single target
body: follow-up https://github.com/scikit-learn/scikit-learn/pull/20355

Check the predictions shape consistency with `y` of shape (n_samples,) and (n_samples, 1).

The list of passing/failing estimators is shown below. I excluded the SearchCV and Pipeline since they depend on the inner estimators.

<details>

```bash
sklearn/tests/test_common.py::test_estimators[ARDRegression()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[AdaBoostRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[BaggingRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[BayesianRidge()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[CCA()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[DecisionTreeRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[DummyRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[ElasticNet()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[ElasticNetCV()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[ExtraTreeRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[ExtraTreesRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[GammaRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[GaussianProcessRegressor()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[GradientBoostingRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[HistGradientBoostingRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[HuberRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[KNeighborsRegressor()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[KernelRidge()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[Lars()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[LarsCV()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[Lasso()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[LassoCV()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[LassoLars()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[LassoLarsCV()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[LassoLarsIC()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[LinearRegression()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[LinearSVR()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[MLPRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[MultiOutputRegressor(estimator=Ridge())-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[MultiTaskElasticNet()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[MultiTaskElasticNetCV()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[MultiTaskLasso()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[MultiTaskLassoCV()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[NuSVR()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[OrthogonalMatchingPursuit()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[OrthogonalMatchingPursuitCV()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[PLSCanonical()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[PLSRegression()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[PassiveAggressiveRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[PoissonRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[QuantileRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[RANSACRegressor(base_estimator=Ridge())-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[RadiusNeighborsRegressor()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[RandomForestRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[RegressorChain(base_estimator=Ridge())-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[Ridge()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[RidgeCV(alphas=array([0.1,1.,10.]))-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[SGDRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[SVR()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[StackingRegressor(estimators=[('est1',Ridge(alpha=0.1)),('est2',Ridge(alpha=1))])-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[TheilSenRegressor(max_subpopulation=10000)-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[TransformedTargetRegressor()-check_regressors_predict_single_target] FAILED
sklearn/tests/test_common.py::test_estimators[TweedieRegressor()-check_regressors_predict_single_target] PASSED
sklearn/tests/test_common.py::test_estimators[VotingRegressor(estimators=[('est1',Ridge(alpha=0.1)),('est2',Ridge(alpha=1))])-check_regressors_predict_single_target] PASSED
```

</details>

The full traceback is available here

<details>

```pytb
====================================================================== FAILURES =======================================================================
____________________________________________ test_estimators[CCA()-check_regressors_predict_single_target] ____________________________________________

estimator = CCA(max_iter=5, n_components=1), check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'CCA')
request = <FixtureRequest for <Function test_estimators[CCA()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'CCA')
estimator  = CCA(max_iter=5, n_components=1)
request    = <FixtureRequest for <Function test_estimators[CCA()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'CCA', regressor_orig = CCA(max_iter=5, n_components=1)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of CCA is not consistent with y.shape=(200,). We expect a shape of (200,) but predictions have a  shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'CCA'
regressor  = CCA(max_iter=5, n_components=1)
regressor_orig = CCA(max_iter=5, n_components=1)
y          = array([ 1.44842059,  0.02462946,  0.17266373, -0.4567032 , -1.1064117 ,
        0.52156598,  0.39521595,  1.45716779, ...090363,  0.4696285 , -0.05612368,  0.30389789,
       -0.96880098, -1.20408431,  0.25088783,  0.4746183 , -1.11748681])
y_pred     = array([[ 0.70341857],
       [ 0.41580605],
       [ 0.51516086],
       [-0.17535823],
       [-0.96079336],
       [...915966],
       [-0.75587075],
       [-1.27098298],
       [ 0.38843129],
       [-0.37049613],
       [-1.46128665]])

sklearn/utils/estimator_checks.py:2584: AssertionError
_________________________________ test_estimators[GaussianProcessRegressor()-check_regressors_predict_single_target] __________________________________

estimator = GaussianProcessRegressor()
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'GaussianProcessRegressor')
request = <FixtureRequest for <Function test_estimators[GaussianProcessRegressor()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'GaussianProcessRegressor')
estimator  = GaussianProcessRegressor()
request    = <FixtureRequest for <Function test_estimators[GaussianProcessRegressor()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'GaussianProcessRegressor', regressor_orig = GaussianProcessRegressor()

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of GaussianProcessRegressor is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'GaussianProcessRegressor'
regressor  = GaussianProcessRegressor(alpha=0.01, random_state=0)
regressor_orig = GaussianProcessRegressor()
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 1.43607283],
       [ 0.02564265],
       [ 0.17168575],
       [-0.45126504],
       [-1.09692214],
       [...113976],
       [-0.95972214],
       [-1.19334259],
       [ 0.24854972],
       [ 0.46966078],
       [-1.10717077]])

sklearn/utils/estimator_checks.py:2593: AssertionError
____________________________________ test_estimators[KNeighborsRegressor()-check_regressors_predict_single_target] ____________________________________

estimator = KNeighborsRegressor(), check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'KNeighborsRegressor')
request = <FixtureRequest for <Function test_estimators[KNeighborsRegressor()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'KNeighborsRegressor')
estimator  = KNeighborsRegressor()
request    = <FixtureRequest for <Function test_estimators[KNeighborsRegressor()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'KNeighborsRegressor', regressor_orig = KNeighborsRegressor()

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of KNeighborsRegressor is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'KNeighborsRegressor'
regressor  = KNeighborsRegressor()
regressor_orig = KNeighborsRegressor()
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 7.03188092e-01],
       [ 4.98803882e-01],
       [ 4.34489736e-01],
       [-2.11787597e-02],
       [-5.132...09349650e-01],
       [-5.51305137e-01],
       [ 5.47142438e-01],
       [ 2.57795837e-01],
       [-9.97464138e-01]])

sklearn/utils/estimator_checks.py:2593: AssertionError
________________________________________ test_estimators[KernelRidge()-check_regressors_predict_single_target] ________________________________________

estimator = KernelRidge(), check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'KernelRidge')
request = <FixtureRequest for <Function test_estimators[KernelRidge()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'KernelRidge')
estimator  = KernelRidge()
request    = <FixtureRequest for <Function test_estimators[KernelRidge()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'KernelRidge', regressor_orig = KernelRidge()

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of KernelRidge is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'KernelRidge'
regressor  = KernelRidge(alpha=0.01)
regressor_orig = KernelRidge()
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 0.64050253],
       [ 0.37861728],
       [ 0.469093  ],
       [-0.15967103],
       [-0.87485222],
       [...631252],
       [-0.68826007],
       [-1.15730675],
       [ 0.35371003],
       [-0.33734535],
       [-1.33060375]])

sklearn/utils/estimator_checks.py:2593: AssertionError
_____________________________________ test_estimators[LinearRegression()-check_regressors_predict_single_target] ______________________________________

estimator = LinearRegression(), check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'LinearRegression')
request = <FixtureRequest for <Function test_estimators[LinearRegression()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'LinearRegression')
estimator  = LinearRegression()
request    = <FixtureRequest for <Function test_estimators[LinearRegression()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'LinearRegression', regressor_orig = LinearRegression()

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of LinearRegression is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'LinearRegression'
regressor  = LinearRegression()
regressor_orig = LinearRegression()
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 0.64053854],
       [ 0.37863629],
       [ 0.46910957],
       [-0.1596826 ],
       [-0.87490606],
       [...63266 ],
       [-0.68830191],
       [-1.1573672 ],
       [ 0.35370862],
       [-0.33737672],
       [-1.33065924]])

sklearn/utils/estimator_checks.py:2593: AssertionError
___________________________ test_estimators[MultiOutputRegressor(estimator=Ridge())-check_regressors_predict_single_target] ___________________________

estimator = MultiOutputRegressor(estimator=Ridge())
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'MultiOutputRegressor')
request = <FixtureRequest for <Function test_estimators[MultiOutputRegressor(estimator=Ridge())-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'MultiOutputRegressor')
estimator  = MultiOutputRegressor(estimator=Ridge())
request    = <FixtureRequest for <Function test_estimators[MultiOutputRegressor(estimator=Ridge())-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'MultiOutputRegressor', regressor_orig = MultiOutputRegressor(estimator=Ridge())

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
>       assert y.ndim == 1
E       AssertionError

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'MultiOutputRegressor'
regressor  = MultiOutputRegressor(estimator=Ridge())
regressor_orig = MultiOutputRegressor(estimator=Ridge())
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])

sklearn/utils/estimator_checks.py:2573: AssertionError
____________________________________ test_estimators[MultiTaskElasticNet()-check_regressors_predict_single_target] ____________________________________

estimator = MultiTaskElasticNet(max_iter=5)
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'MultiTaskElasticNet')
request = <FixtureRequest for <Function test_estimators[MultiTaskElasticNet()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'MultiTaskElasticNet')
estimator  = MultiTaskElasticNet(max_iter=5)
request    = <FixtureRequest for <Function test_estimators[MultiTaskElasticNet()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'MultiTaskElasticNet', regressor_orig = MultiTaskElasticNet(max_iter=5)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
>       assert y.ndim == 1
E       AssertionError

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'MultiTaskElasticNet'
regressor  = MultiTaskElasticNet(max_iter=5)
regressor_orig = MultiTaskElasticNet(max_iter=5)
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])

sklearn/utils/estimator_checks.py:2573: AssertionError
___________________________________ test_estimators[MultiTaskElasticNetCV()-check_regressors_predict_single_target] ___________________________________

estimator = MultiTaskElasticNetCV(cv=3, max_iter=5)
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'MultiTaskElasticNetCV')
request = <FixtureRequest for <Function test_estimators[MultiTaskElasticNetCV()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'MultiTaskElasticNetCV')
estimator  = MultiTaskElasticNetCV(cv=3, max_iter=5)
request    = <FixtureRequest for <Function test_estimators[MultiTaskElasticNetCV()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'MultiTaskElasticNetCV', regressor_orig = MultiTaskElasticNetCV(cv=3, max_iter=5)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
>       assert y.ndim == 1
E       AssertionError

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'MultiTaskElasticNetCV'
regressor  = MultiTaskElasticNetCV(cv=3, max_iter=5)
regressor_orig = MultiTaskElasticNetCV(cv=3, max_iter=5)
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])

sklearn/utils/estimator_checks.py:2573: AssertionError
______________________________________ test_estimators[MultiTaskLasso()-check_regressors_predict_single_target] _______________________________________

estimator = MultiTaskLasso(max_iter=5), check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'MultiTaskLasso')
request = <FixtureRequest for <Function test_estimators[MultiTaskLasso()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'MultiTaskLasso')
estimator  = MultiTaskLasso(max_iter=5)
request    = <FixtureRequest for <Function test_estimators[MultiTaskLasso()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'MultiTaskLasso', regressor_orig = MultiTaskLasso(max_iter=5)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
>       assert y.ndim == 1
E       AssertionError

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'MultiTaskLasso'
regressor  = MultiTaskLasso(max_iter=5)
regressor_orig = MultiTaskLasso(max_iter=5)
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])

sklearn/utils/estimator_checks.py:2573: AssertionError
_____________________________________ test_estimators[MultiTaskLassoCV()-check_regressors_predict_single_target] ______________________________________

estimator = MultiTaskLassoCV(cv=3, max_iter=5)
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'MultiTaskLassoCV')
request = <FixtureRequest for <Function test_estimators[MultiTaskLassoCV()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'MultiTaskLassoCV')
estimator  = MultiTaskLassoCV(cv=3, max_iter=5)
request    = <FixtureRequest for <Function test_estimators[MultiTaskLassoCV()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'MultiTaskLassoCV', regressor_orig = MultiTaskLassoCV(cv=3, max_iter=5)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
>       assert y.ndim == 1
E       AssertionError

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'MultiTaskLassoCV'
regressor  = MultiTaskLassoCV(cv=3, max_iter=5)
regressor_orig = MultiTaskLassoCV(cv=3, max_iter=5)
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])

sklearn/utils/estimator_checks.py:2573: AssertionError
_______________________________________ test_estimators[PLSCanonical()-check_regressors_predict_single_target] ________________________________________

estimator = PLSCanonical(max_iter=5, n_components=1)
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'PLSCanonical')
request = <FixtureRequest for <Function test_estimators[PLSCanonical()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'PLSCanonical')
estimator  = PLSCanonical(max_iter=5, n_components=1)
request    = <FixtureRequest for <Function test_estimators[PLSCanonical()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'PLSCanonical', regressor_orig = PLSCanonical(max_iter=5, n_components=1)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of PLSCanonical is not consistent with y.shape=(200,). We expect a shape of (200,) but predictions have a  shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'PLSCanonical'
regressor  = PLSCanonical(max_iter=5, n_components=1)
regressor_orig = PLSCanonical(max_iter=5, n_components=1)
y          = array([ 1.44842059,  0.02462946,  0.17266373, -0.4567032 , -1.1064117 ,
        0.52156598,  0.39521595,  1.45716779, ...090363,  0.4696285 , -0.05612368,  0.30389789,
       -0.96880098, -1.20408431,  0.25088783,  0.4746183 , -1.11748681])
y_pred     = array([[ 0.6060054 ],
       [ 0.47097687],
       [ 0.67081485],
       [-0.11044873],
       [-0.75074555],
       [...252769],
       [-0.67251118],
       [-1.28084969],
       [ 0.80332846],
       [-0.08654241],
       [-1.80608003]])

sklearn/utils/estimator_checks.py:2584: AssertionError
_______________________________________ test_estimators[PLSRegression()-check_regressors_predict_single_target] _______________________________________

estimator = PLSRegression(max_iter=5, n_components=1)
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'PLSRegression')
request = <FixtureRequest for <Function test_estimators[PLSRegression()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'PLSRegression')
estimator  = PLSRegression(max_iter=5, n_components=1)
request    = <FixtureRequest for <Function test_estimators[PLSRegression()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'PLSRegression', regressor_orig = PLSRegression(max_iter=5, n_components=1)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of PLSRegression is not consistent with y.shape=(200,). We expect a shape of (200,) but predictions have a  shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'PLSRegression'
regressor  = PLSRegression(max_iter=5, n_components=1)
regressor_orig = PLSRegression(max_iter=5, n_components=1)
y          = array([ 1.44842059,  0.02462946,  0.17266373, -0.4567032 , -1.1064117 ,
        0.52156598,  0.39521595,  1.45716779, ...090363,  0.4696285 , -0.05612368,  0.30389789,
       -0.96880098, -1.20408431,  0.25088783,  0.4746183 , -1.11748681])
y_pred     = array([[ 0.51363173],
       [ 0.39918566],
       [ 0.56856225],
       [-0.09361298],
       [-0.63630908],
       [...0014  ],
       [-0.57000001],
       [-1.08560921],
       [ 0.68087675],
       [-0.07335071],
       [-1.53077847]])

sklearn/utils/estimator_checks.py:2584: AssertionError
___________________________ test_estimators[RANSACRegressor(base_estimator=Ridge())-check_regressors_predict_single_target] ___________________________

estimator = RANSACRegressor(base_estimator=Ridge(), max_trials=10)
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'RANSACRegressor')
request = <FixtureRequest for <Function test_estimators[RANSACRegressor(base_estimator=Ridge())-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'RANSACRegressor')
estimator  = RANSACRegressor(base_estimator=Ridge(), max_trials=10)
request    = <FixtureRequest for <Function test_estimators[RANSACRegressor(base_estimator=Ridge())-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'RANSACRegressor', regressor_orig = RANSACRegressor(base_estimator=Ridge(), max_trials=10)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of RANSACRegressor is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'RANSACRegressor'
regressor  = RANSACRegressor(base_estimator=Ridge(), max_trials=10, random_state=0)
regressor_orig = RANSACRegressor(base_estimator=Ridge(), max_trials=10)
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 5.37741736e-01],
       [ 4.33413224e-01],
       [ 2.59363700e-01],
       [-5.57968654e-02],
       [-4.712...25876395e-01],
       [-1.03453610e+00],
       [ 8.47355797e-01],
       [ 2.44500362e-01],
       [-1.40208628e+00]])

sklearn/utils/estimator_checks.py:2593: AssertionError
_________________________________ test_estimators[RadiusNeighborsRegressor()-check_regressors_predict_single_target] __________________________________

estimator = RadiusNeighborsRegressor()
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'RadiusNeighborsRegressor')
request = <FixtureRequest for <Function test_estimators[RadiusNeighborsRegressor()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'RadiusNeighborsRegressor')
estimator  = RadiusNeighborsRegressor()
request    = <FixtureRequest for <Function test_estimators[RadiusNeighborsRegressor()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'RadiusNeighborsRegressor', regressor_orig = RadiusNeighborsRegressor()

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of RadiusNeighborsRegressor is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'RadiusNeighborsRegressor'
regressor  = RadiusNeighborsRegressor()
regressor_orig = RadiusNeighborsRegressor()
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])

sklearn/utils/estimator_checks.py:2593: AssertionError
___________________________ test_estimators[RegressorChain(base_estimator=Ridge())-check_regressors_predict_single_target] ____________________________

estimator = RegressorChain(base_estimator=Ridge(), cv=3)
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'RegressorChain')
request = <FixtureRequest for <Function test_estimators[RegressorChain(base_estimator=Ridge())-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'RegressorChain')
estimator  = RegressorChain(base_estimator=Ridge(), cv=3)
request    = <FixtureRequest for <Function test_estimators[RegressorChain(base_estimator=Ridge())-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'RegressorChain', regressor_orig = RegressorChain(base_estimator=Ridge(), cv=3)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
>       assert y.ndim == 1
E       AssertionError

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'RegressorChain'
regressor  = RegressorChain(base_estimator=Ridge(), cv=3)
regressor_orig = RegressorChain(base_estimator=Ridge(), cv=3)
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])

sklearn/utils/estimator_checks.py:2573: AssertionError
___________________________________________ test_estimators[Ridge()-check_regressors_predict_single_target] ___________________________________________

estimator = Ridge(), check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'Ridge')
request = <FixtureRequest for <Function test_estimators[Ridge()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'Ridge')
estimator  = Ridge()
request    = <FixtureRequest for <Function test_estimators[Ridge()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'Ridge', regressor_orig = Ridge()

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of Ridge is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'Ridge'
regressor  = Ridge(alpha=0.01, random_state=0)
regressor_orig = Ridge()
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 0.64050253],
       [ 0.37861728],
       [ 0.469093  ],
       [-0.15967103],
       [-0.87485222],
       [...631252],
       [-0.68826007],
       [-1.15730675],
       [ 0.35371003],
       [-0.33734535],
       [-1.33060375]])

sklearn/utils/estimator_checks.py:2593: AssertionError
_____________________________ test_estimators[RidgeCV(alphas=array([0.1,1.,10.]))-check_regressors_predict_single_target] _____________________________

estimator = RidgeCV(alphas=array([ 0.1,  1. , 10. ]))
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'RidgeCV')
request = <FixtureRequest for <Function test_estimators[RidgeCV(alphas=array([0.1,1.,10.]))-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'RidgeCV')
estimator  = RidgeCV(alphas=array([ 0.1,  1. , 10. ]))
request    = <FixtureRequest for <Function test_estimators[RidgeCV(alphas=array([0.1,1.,10.]))-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'RidgeCV', regressor_orig = RidgeCV(alphas=array([ 0.1,  1. , 10. ]))

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of RidgeCV is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'RidgeCV'
regressor  = RidgeCV(alphas=array([ 0.1,  1. , 10. ]))
regressor_orig = RidgeCV(alphas=array([ 0.1,  1. , 10. ]))
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 0.63695684],
       [ 0.37674617],
       [ 0.46745671],
       [-0.15853397],
       [-0.8695535 ],
       [...492207],
       [-0.68414443],
       [-1.15135559],
       [ 0.35383855],
       [-0.33426329],
       [-1.32513407]])

sklearn/utils/estimator_checks.py:2593: AssertionError
________________________________ test_estimators[TransformedTargetRegressor()-check_regressors_predict_single_target] _________________________________

estimator = TransformedTargetRegressor()
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'TransformedTargetRegressor')
request = <FixtureRequest for <Function test_estimators[TransformedTargetRegressor()-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(
            category=(FutureWarning, ConvergenceWarning, UserWarning, FutureWarning)
        ):
            _set_checking_parameters(estimator)
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'TransformedTargetRegressor')
estimator  = TransformedTargetRegressor()
request    = <FixtureRequest for <Function test_estimators[TransformedTargetRegressor()-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'TransformedTargetRegressor', regressor_orig = TransformedTargetRegressor()

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of TransformedTargetRegressor is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'TransformedTargetRegressor'
regressor  = TransformedTargetRegressor()
regressor_orig = TransformedTargetRegressor()
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 0.64053854],
       [ 0.37863629],
       [ 0.46910957],
       [-0.1596826 ],
       [-0.87490606],
       [...63266 ],
       [-0.68830191],
       [-1.1573672 ],
       [ 0.35370862],
       [-0.33737672],
       [-1.33065924]])

sklearn/utils/estimator_checks.py:2593: AssertionError
_____________ test_search_cv[GridSearchCV(cv=2,estimator=Ridge(),param_grid={'alpha':[0.1,1.0]})-check_regressors_predict_single_target] ______________

estimator = GridSearchCV(cv=2, estimator=Ridge(), param_grid={'alpha': [0.1, 1.0]})
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'GridSearchCV')
request = <FixtureRequest for <Function test_search_cv[GridSearchCV(cv=2,estimator=Ridge(),param_grid={'alpha':[0.1,1.0]})-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_generate_search_cv_instances()))
    def test_search_cv(estimator, check, request):
        # Common tests for SearchCV instances
        # We have a separate test because those meta-estimators can accept a
        # wide range of base estimators (classifiers, regressors, pipelines)
        with ignore_warnings(
            category=(
                FutureWarning,
                ConvergenceWarning,
                UserWarning,
                FutureWarning,
                FitFailedWarning,
            )
        ):
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'GridSearchCV')
estimator  = GridSearchCV(cv=2, estimator=Ridge(), param_grid={'alpha': [0.1, 1.0]})
request    = <FixtureRequest for <Function test_search_cv[GridSearchCV(cv=2,estimator=Ridge(),param_grid={'alpha':[0.1,1.0]})-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:301: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'GridSearchCV', regressor_orig = GridSearchCV(cv=2, estimator=Ridge(), param_grid={'alpha': [0.1, 1.0]})

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of GridSearchCV is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'GridSearchCV'
regressor  = GridSearchCV(cv=2, estimator=Ridge(), param_grid={'alpha': [0.1, 1.0]})
regressor_orig = GridSearchCV(cv=2, estimator=Ridge(), param_grid={'alpha': [0.1, 1.0]})
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 0.63695684],
       [ 0.37674617],
       [ 0.46745671],
       [-0.15853397],
       [-0.8695535 ],
       [...492207],
       [-0.68414443],
       [-1.15135559],
       [ 0.35383855],
       [-0.33426329],
       [-1.32513407]])

sklearn/utils/estimator_checks.py:2593: AssertionError
_ test_search_cv[HalvingGridSearchCV(cv=2,estimator=Ridge(),min_resources='smallest',param_grid={'alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target0] _

estimator = HalvingGridSearchCV(cv=2, estimator=Ridge(), min_resources='smallest',
                    param_grid={'alpha': [0.1, 1.0]}, random_state=0)
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'HalvingGridSearchCV')
request = <FixtureRequest for <Function test_search_cv[HalvingGridSearchCV(cv=2,estimator=Ridge(),min_resources='smallest',param_grid={'alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target0]>>

    @parametrize_with_checks(list(_generate_search_cv_instances()))
    def test_search_cv(estimator, check, request):
        # Common tests for SearchCV instances
        # We have a separate test because those meta-estimators can accept a
        # wide range of base estimators (classifiers, regressors, pipelines)
        with ignore_warnings(
            category=(
                FutureWarning,
                ConvergenceWarning,
                UserWarning,
                FutureWarning,
                FitFailedWarning,
            )
        ):
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'HalvingGridSearchCV')
estimator  = HalvingGridSearchCV(cv=2, estimator=Ridge(), min_resources='smallest',
                    param_grid={'alpha': [0.1, 1.0]}, random_state=0)
request    = <FixtureRequest for <Function test_search_cv[HalvingGridSearchCV(cv=2,estimator=Ridge(),min_resources='smallest',param_grid={'alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target0]>>

sklearn/tests/test_common.py:301: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'HalvingGridSearchCV'
regressor_orig = HalvingGridSearchCV(cv=2, estimator=Ridge(), min_resources='smallest',
                    param_grid={'alpha': [0.1, 1.0]}, random_state=0)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of HalvingGridSearchCV is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'HalvingGridSearchCV'
regressor  = HalvingGridSearchCV(cv=2, estimator=Ridge(), min_resources='smallest',
                    param_grid={'alpha': [0.1, 1.0]}, random_state=0)
regressor_orig = HalvingGridSearchCV(cv=2, estimator=Ridge(), min_resources='smallest',
                    param_grid={'alpha': [0.1, 1.0]}, random_state=0)
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 0.63695684],
       [ 0.37674617],
       [ 0.46745671],
       [-0.15853397],
       [-0.8695535 ],
       [...492207],
       [-0.68414443],
       [-1.15135559],
       [ 0.35383855],
       [-0.33426329],
       [-1.32513407]])

sklearn/utils/estimator_checks.py:2593: AssertionError
_ test_search_cv[RandomizedSearchCV(cv=2,estimator=Ridge(),param_distributions={'alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target] _

estimator = RandomizedSearchCV(cv=2, estimator=Ridge(),
                   param_distributions={'alpha': [0.1, 1.0]}, random_state=0)
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'RandomizedSearchCV')
request = <FixtureRequest for <Function test_search_cv[RandomizedSearchCV(cv=2,estimator=Ridge(),param_distributions={'alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_generate_search_cv_instances()))
    def test_search_cv(estimator, check, request):
        # Common tests for SearchCV instances
        # We have a separate test because those meta-estimators can accept a
        # wide range of base estimators (classifiers, regressors, pipelines)
        with ignore_warnings(
            category=(
                FutureWarning,
                ConvergenceWarning,
                UserWarning,
                FutureWarning,
                FitFailedWarning,
            )
        ):
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'RandomizedSearchCV')
estimator  = RandomizedSearchCV(cv=2, estimator=Ridge(),
                   param_distributions={'alpha': [0.1, 1.0]}, random_state=0)
request    = <FixtureRequest for <Function test_search_cv[RandomizedSearchCV(cv=2,estimator=Ridge(),param_distributions={'alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:301: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'RandomizedSearchCV'
regressor_orig = RandomizedSearchCV(cv=2, estimator=Ridge(),
                   param_distributions={'alpha': [0.1, 1.0]}, random_state=0)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of RandomizedSearchCV is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'RandomizedSearchCV'
regressor  = RandomizedSearchCV(cv=2, estimator=Ridge(),
                   param_distributions={'alpha': [0.1, 1.0]}, random_state=0)
regressor_orig = RandomizedSearchCV(cv=2, estimator=Ridge(),
                   param_distributions={'alpha': [0.1, 1.0]}, random_state=0)
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 0.63695684],
       [ 0.37674617],
       [ 0.46745671],
       [-0.15853397],
       [-0.8695535 ],
       [...492207],
       [-0.68414443],
       [-1.15135559],
       [ 0.35383855],
       [-0.33426329],
       [-1.32513407]])

sklearn/utils/estimator_checks.py:2593: AssertionError
_ test_search_cv[HalvingGridSearchCV(cv=2,estimator=Ridge(),min_resources='smallest',param_grid={'alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target1] _

estimator = HalvingGridSearchCV(cv=2, estimator=Ridge(), min_resources='smallest',
                    param_grid={'alpha': [0.1, 1.0]}, random_state=0)
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'HalvingGridSearchCV')
request = <FixtureRequest for <Function test_search_cv[HalvingGridSearchCV(cv=2,estimator=Ridge(),min_resources='smallest',param_grid={'alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target1]>>

    @parametrize_with_checks(list(_generate_search_cv_instances()))
    def test_search_cv(estimator, check, request):
        # Common tests for SearchCV instances
        # We have a separate test because those meta-estimators can accept a
        # wide range of base estimators (classifiers, regressors, pipelines)
        with ignore_warnings(
            category=(
                FutureWarning,
                ConvergenceWarning,
                UserWarning,
                FutureWarning,
                FitFailedWarning,
            )
        ):
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'HalvingGridSearchCV')
estimator  = HalvingGridSearchCV(cv=2, estimator=Ridge(), min_resources='smallest',
                    param_grid={'alpha': [0.1, 1.0]}, random_state=0)
request    = <FixtureRequest for <Function test_search_cv[HalvingGridSearchCV(cv=2,estimator=Ridge(),min_resources='smallest',param_grid={'alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target1]>>

sklearn/tests/test_common.py:301: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'HalvingGridSearchCV'
regressor_orig = HalvingGridSearchCV(cv=2, estimator=Ridge(), min_resources='smallest',
                    param_grid={'alpha': [0.1, 1.0]}, random_state=0)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of HalvingGridSearchCV is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'HalvingGridSearchCV'
regressor  = HalvingGridSearchCV(cv=2, estimator=Ridge(), min_resources='smallest',
                    param_grid={'alpha': [0.1, 1.0]}, random_state=0)
regressor_orig = HalvingGridSearchCV(cv=2, estimator=Ridge(), min_resources='smallest',
                    param_grid={'alpha': [0.1, 1.0]}, random_state=0)
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 0.63695684],
       [ 0.37674617],
       [ 0.46745671],
       [-0.15853397],
       [-0.8695535 ],
       [...492207],
       [-0.68414443],
       [-1.15135559],
       [ 0.35383855],
       [-0.33426329],
       [-1.32513407]])

sklearn/utils/estimator_checks.py:2593: AssertionError
_ test_search_cv[GridSearchCV(cv=2,error_score='raise',estimator=Pipeline(steps=[('pca',PCA()),('ridge',Ridge())]),param_grid={'ridge__alpha':[0.1,1.0]})-check_regressors_predict_single_target] _

estimator = GridSearchCV(cv=2, error_score='raise',
             estimator=Pipeline(steps=[('pca', PCA()), ('ridge', Ridge())]),
             param_grid={'ridge__alpha': [0.1, 1.0]})
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'GridSearchCV')
request = <FixtureRequest for <Function test_search_cv[GridSearchCV(cv=2,error_score='raise',estimator=Pipeline(steps=[('pca',PCA()),('ridge',Ridge())]),param_grid={'ridge__alpha':[0.1,1.0]})-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_generate_search_cv_instances()))
    def test_search_cv(estimator, check, request):
        # Common tests for SearchCV instances
        # We have a separate test because those meta-estimators can accept a
        # wide range of base estimators (classifiers, regressors, pipelines)
        with ignore_warnings(
            category=(
                FutureWarning,
                ConvergenceWarning,
                UserWarning,
                FutureWarning,
                FitFailedWarning,
            )
        ):
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'GridSearchCV')
estimator  = GridSearchCV(cv=2, error_score='raise',
             estimator=Pipeline(steps=[('pca', PCA()), ('ridge', Ridge())]),
             param_grid={'ridge__alpha': [0.1, 1.0]})
request    = <FixtureRequest for <Function test_search_cv[GridSearchCV(cv=2,error_score='raise',estimator=Pipeline(steps=[('pca',PCA()),('ridge',Ridge())]),param_grid={'ridge__alpha':[0.1,1.0]})-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:301: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'GridSearchCV'
regressor_orig = GridSearchCV(cv=2, error_score='raise',
             estimator=Pipeline(steps=[('pca', PCA()), ('ridge', Ridge())]),
             param_grid={'ridge__alpha': [0.1, 1.0]})

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of GridSearchCV is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'GridSearchCV'
regressor  = GridSearchCV(cv=2, error_score='raise',
             estimator=Pipeline(steps=[('pca', PCA()), ('ridge', Ridge())]),
             param_grid={'ridge__alpha': [0.1, 1.0]})
regressor_orig = GridSearchCV(cv=2, error_score='raise',
             estimator=Pipeline(steps=[('pca', PCA()), ('ridge', Ridge())]),
             param_grid={'ridge__alpha': [0.1, 1.0]})
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 0.63695684],
       [ 0.37674617],
       [ 0.46745671],
       [-0.15853397],
       [-0.8695535 ],
       [...492207],
       [-0.68414443],
       [-1.15135559],
       [ 0.35383855],
       [-0.33426329],
       [-1.32513407]])

sklearn/utils/estimator_checks.py:2593: AssertionError
_ test_search_cv[HalvingGridSearchCV(cv=2,error_score='raise',estimator=Pipeline(steps=[('pca',PCA()),('ridge',Ridge())]),min_resources='smallest',param_grid={'ridge__alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target] _

estimator = HalvingGridSearchCV(cv=2, error_score='raise',
                    estimator=Pipeline(steps=[('pca', PCA()),
         ...                 min_resources='smallest',
                    param_grid={'ridge__alpha': [0.1, 1.0]}, random_state=0)
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'HalvingGridSearchCV')
request = <FixtureRequest for <Function test_search_cv[HalvingGridSearchCV(cv=2,error_score='raise',estimator=Pipeline(steps=[('...in_resources='smallest',param_grid={'ridge__alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_generate_search_cv_instances()))
    def test_search_cv(estimator, check, request):
        # Common tests for SearchCV instances
        # We have a separate test because those meta-estimators can accept a
        # wide range of base estimators (classifiers, regressors, pipelines)
        with ignore_warnings(
            category=(
                FutureWarning,
                ConvergenceWarning,
                UserWarning,
                FutureWarning,
                FitFailedWarning,
            )
        ):
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'HalvingGridSearchCV')
estimator  = HalvingGridSearchCV(cv=2, error_score='raise',
                    estimator=Pipeline(steps=[('pca', PCA()),
         ...                 min_resources='smallest',
                    param_grid={'ridge__alpha': [0.1, 1.0]}, random_state=0)
request    = <FixtureRequest for <Function test_search_cv[HalvingGridSearchCV(cv=2,error_score='raise',estimator=Pipeline(steps=[('...in_resources='smallest',param_grid={'ridge__alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:301: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'HalvingGridSearchCV'
regressor_orig = HalvingGridSearchCV(cv=2, error_score='raise',
                    estimator=Pipeline(steps=[('pca', PCA()),
         ...                 min_resources='smallest',
                    param_grid={'ridge__alpha': [0.1, 1.0]}, random_state=0)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of HalvingGridSearchCV is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'HalvingGridSearchCV'
regressor  = HalvingGridSearchCV(cv=2, error_score='raise',
                    estimator=Pipeline(steps=[('pca', PCA()),
         ...                 min_resources='smallest',
                    param_grid={'ridge__alpha': [0.1, 1.0]}, random_state=0)
regressor_orig = HalvingGridSearchCV(cv=2, error_score='raise',
                    estimator=Pipeline(steps=[('pca', PCA()),
         ...                 min_resources='smallest',
                    param_grid={'ridge__alpha': [0.1, 1.0]}, random_state=0)
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 0.63695684],
       [ 0.37674617],
       [ 0.46745671],
       [-0.15853397],
       [-0.8695535 ],
       [...492207],
       [-0.68414443],
       [-1.15135559],
       [ 0.35383855],
       [-0.33426329],
       [-1.32513407]])

sklearn/utils/estimator_checks.py:2593: AssertionError
_ test_search_cv[RandomizedSearchCV(cv=2,error_score='raise',estimator=Pipeline(steps=[('pca',PCA()),('ridge',Ridge())]),param_distributions={'ridge__alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target] _

estimator = RandomizedSearchCV(cv=2, error_score='raise',
                   estimator=Pipeline(steps=[('pca', PCA()),
           ...e', Ridge())]),
                   param_distributions={'ridge__alpha': [0.1, 1.0]},
                   random_state=0)
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'RandomizedSearchCV')
request = <FixtureRequest for <Function test_search_cv[RandomizedSearchCV(cv=2,error_score='raise',estimator=Pipeline(steps=[('p...ge',Ridge())]),param_distributions={'ridge__alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_generate_search_cv_instances()))
    def test_search_cv(estimator, check, request):
        # Common tests for SearchCV instances
        # We have a separate test because those meta-estimators can accept a
        # wide range of base estimators (classifiers, regressors, pipelines)
        with ignore_warnings(
            category=(
                FutureWarning,
                ConvergenceWarning,
                UserWarning,
                FutureWarning,
                FitFailedWarning,
            )
        ):
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'RandomizedSearchCV')
estimator  = RandomizedSearchCV(cv=2, error_score='raise',
                   estimator=Pipeline(steps=[('pca', PCA()),
           ...e', Ridge())]),
                   param_distributions={'ridge__alpha': [0.1, 1.0]},
                   random_state=0)
request    = <FixtureRequest for <Function test_search_cv[RandomizedSearchCV(cv=2,error_score='raise',estimator=Pipeline(steps=[('p...ge',Ridge())]),param_distributions={'ridge__alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:301: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'RandomizedSearchCV'
regressor_orig = RandomizedSearchCV(cv=2, error_score='raise',
                   estimator=Pipeline(steps=[('pca', PCA()),
           ...e', Ridge())]),
                   param_distributions={'ridge__alpha': [0.1, 1.0]},
                   random_state=0)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of RandomizedSearchCV is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'RandomizedSearchCV'
regressor  = RandomizedSearchCV(cv=2, error_score='raise',
                   estimator=Pipeline(steps=[('pca', PCA()),
           ...e', Ridge())]),
                   param_distributions={'ridge__alpha': [0.1, 1.0]},
                   random_state=0)
regressor_orig = RandomizedSearchCV(cv=2, error_score='raise',
                   estimator=Pipeline(steps=[('pca', PCA()),
           ...e', Ridge())]),
                   param_distributions={'ridge__alpha': [0.1, 1.0]},
                   random_state=0)
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 0.63695684],
       [ 0.37674617],
       [ 0.46745671],
       [-0.15853397],
       [-0.8695535 ],
       [...492207],
       [-0.68414443],
       [-1.15135559],
       [ 0.35383855],
       [-0.33426329],
       [-1.32513407]])

sklearn/utils/estimator_checks.py:2593: AssertionError
_ test_search_cv[HalvingRandomSearchCV(cv=2,error_score='raise',estimator=Pipeline(steps=[('pca',PCA()),('ridge',Ridge())]),param_distributions={'ridge__alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target] _

estimator = HalvingRandomSearchCV(cv=2, error_score='raise',
                      estimator=Pipeline(steps=[('pca', PCA()),
     ...dge())]),
                      param_distributions={'ridge__alpha': [0.1, 1.0]},
                      random_state=0)
check = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'HalvingRandomSearchCV')
request = <FixtureRequest for <Function test_search_cv[HalvingRandomSearchCV(cv=2,error_score='raise',estimator=Pipeline(steps=[...ge',Ridge())]),param_distributions={'ridge__alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target]>>

    @parametrize_with_checks(list(_generate_search_cv_instances()))
    def test_search_cv(estimator, check, request):
        # Common tests for SearchCV instances
        # We have a separate test because those meta-estimators can accept a
        # wide range of base estimators (classifiers, regressors, pipelines)
        with ignore_warnings(
            category=(
                FutureWarning,
                ConvergenceWarning,
                UserWarning,
                FutureWarning,
                FitFailedWarning,
            )
        ):
>           check(estimator)

check      = functools.partial(<function check_regressors_predict_single_target at 0x11dc2d1f0>, 'HalvingRandomSearchCV')
estimator  = HalvingRandomSearchCV(cv=2, error_score='raise',
                      estimator=Pipeline(steps=[('pca', PCA()),
     ...dge())]),
                      param_distributions={'ridge__alpha': [0.1, 1.0]},
                      random_state=0)
request    = <FixtureRequest for <Function test_search_cv[HalvingRandomSearchCV(cv=2,error_score='raise',estimator=Pipeline(steps=[...ge',Ridge())]),param_distributions={'ridge__alpha':[0.1,1.0]},random_state=0)-check_regressors_predict_single_target]>>

sklearn/tests/test_common.py:301: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'HalvingRandomSearchCV'
regressor_orig = HalvingRandomSearchCV(cv=2, error_score='raise',
                      estimator=Pipeline(steps=[('pca', PCA()),
     ...dge())]),
                      param_distributions={'ridge__alpha': [0.1, 1.0]},
                      random_state=0)

    def check_regressors_predict_single_target(name, regressor_orig):
        # check the consistency of the prediction obtained from regressors with
        # single target.
        X, y = _regression_dataset()
        X = _pairwise_estimator_convert_X(X, regressor_orig)
        y = scale(y)  # X is already scaled
        regressor = clone(regressor_orig)
        y = _enforce_estimator_tags_y(regressor, y)
        assert y.ndim == 1
    
        if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
            # linear regressors need to set alpha, but not generalized CV ones
            regressor.alpha = 0.01
        if name == "PassiveAggressiveRegressor":
            regressor.C = 0.01
    
        set_random_state(regressor)
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
        assert y_pred.shape == y.shape, (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {y.shape} but predictions have a "
            f" shape of {y_pred.shape}."
        )
    
        y = y[:, np.newaxis]
        regressor.fit(X, y)
        y_pred = regressor.predict(X)
>       assert y_pred.shape == (y.shape[0],), (
            f"The shape of the prediction of {name} is not consistent with "
            f"y.shape={y.shape}. We expect a shape of {(y.shape[0],)} but predictions have "
            f"a shape of {y_pred.shape}."
        )
E       AssertionError: The shape of the prediction of HalvingRandomSearchCV is not consistent with y.shape=(200, 1). We expect a shape of (200,) but predictions have a shape of (200, 1).

X          = array([[ 0.29213817,  1.31098939, -1.34589423, ..., -0.13099541,
         1.02617223,  0.52005686],
       [-1.4558723...193, -0.91441039],
       [ 2.59455523,  1.14231106, -0.15404749, ..., -0.05455829,
         0.41403923,  0.39284481]])
name       = 'HalvingRandomSearchCV'
regressor  = HalvingRandomSearchCV(cv=2, error_score='raise',
                      estimator=Pipeline(steps=[('pca', PCA()),
     ...dge())]),
                      param_distributions={'ridge__alpha': [0.1, 1.0]},
                      random_state=0)
regressor_orig = HalvingRandomSearchCV(cv=2, error_score='raise',
                      estimator=Pipeline(steps=[('pca', PCA()),
     ...dge())]),
                      param_distributions={'ridge__alpha': [0.1, 1.0]},
                      random_state=0)
y          = array([[ 1.44842059],
       [ 0.02462946],
       [ 0.17266373],
       [-0.4567032 ],
       [-1.1064117 ],
       [...389789],
       [-0.96880098],
       [-1.20408431],
       [ 0.25088783],
       [ 0.4746183 ],
       [-1.11748681]])
y_pred     = array([[ 0.63695684],
       [ 0.37674617],
       [ 0.46745671],
       [-0.15853397],
       [-0.8695535 ],
       [...492207],
       [-0.68414443],
       [-1.15135559],
       [ 0.35383855],
       [-0.33426329],
       [-1.32513407]])

sklearn/utils/estimator_checks.py:2593: AssertionError
```
</details>
user: glemaitre
issue_20595: state: open
title: Parallelization in LocallyLinearEmbedding
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

In [**Locally Linear Embedding**](https://github.com/scikit-learn/scikit-learn/blob/2beed55847ee70d363bdbfe14ee4401438fba057/sklearn/manifold/_spectral_embedding.py), the process of local reconstruction for each of _n_ points (presented in [**barycenter_weights**](https://github.com/scikit-learn/scikit-learn/blob/2beed55847ee70d363bdbfe14ee4401438fba057/sklearn/manifold/_locally_linear.py#L22) function) is the most computationally costly one, and as far as I can see, the loop over the points could be easily parallelized to greatly improve the performance, especially since _n_jobs_ parameter is already utilized in [**barycenter_kneighbors_graph**](https://github.com/scikit-learn/scikit-learn/blob/2beed55847ee70d363bdbfe14ee4401438fba057/sklearn/manifold/_locally_linear.py#L75) which calls **barycenter_weights**.
user: MandaloreUltimate
issue_20591: state: open
title: PCA inconsistency with adding/subtracting mean for transform and inverse_transform when n_components=0
body: 

When using n_components=0 in the PCA module, and using transform followed by inverse_transform, it returns the mean of the data. I am not sure if this is intended behaviour but it looks like bug to me as this is happening because the transform is not subtracting the mean but the inverse_transform is adding the mean.



Example:
```python

import numpy as np
from sklearn.decomposition import PCA
m = np.eye(2)
pca_eye = PCA(n_components=0)
pca_eye.fit(m)
print("Reconstruct", pca_eye.inverse_transform(pca_eye.transform([[1,0]])))
print("Transform", pca_eye.transform([[1,0]]))
print("Inverse transform", pca_eye.inverse_transform([]))
```

#### Expected Results
Ideally transform and inverse_transform should behave consistently. I would expect either them both to subtract and add the mean respectively or none.

#### Actual Results
```
Reconstruct [[0.5 0.5]]
Transform []
Inverse transform [0.5 0.5]
```

#### Versions
```
System:
    python: 3.7.0 (default, Oct  9 2018, 10:31:47)  [GCC 7.3.0]
executable: /homes/53/amartya/miniconda3/envs/sneaky/bin/python
   machine: Linux-4.18.0-147.8.1.el8_1.x86_64-x86_64-with-centos-8.1.1911-Core

Python dependencies:
          pip: 21.1.2
   setuptools: 52.0.0.post20210125
      sklearn: 0.24.2
        numpy: 1.19.2
        scipy: 1.6.2
       Cython: None
       pandas: 1.2.4
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```

Perhaps this is intended behaviour but it seems a bit counterintuitive  to me.

user: amartya18x
labels: Needs Decision - API: Requires decision regarding API
Needs Decision - Backward Compatibility: Requires decision regarding backward compatibility
issue_20585: state: open
title: RFC Private attributes copied in `clone`
body: #20350 is an implementation of [SLEP006](https://scikit-learn-enhancement-proposals.readthedocs.io/en/latest/slep006/proposal.html), and it introduces a few methods (e.g. `fit_requests`) which set routing data for the metadata which is then handled by meta-estimators such as `Pipeline` or `GridSearchCV`. 

Those routing information are stored in a `_metadata_request` instance attribute which needs to be copied during `clone`.

At the moment, #20350 has `_metadata_request` as a hard coded attribute which is copied during `clone`, and the following line is added to `check_no_attributes_set_in_init` to ignore private attributes:

``` python
    invalid_attr = set([attr for attr in invalid_attr if not attr.startswith("_")])
```

My suggestion is to register private attributes which need to be copied in `clone` as an estimator tag, and ignore those attributes in the `check_no_attributes_set_in_init` test.

We could also change the test and explicitly only test the `__init__` instead of testing it through `clone`, which means we would still require estimators not to set anything during `__init__`.

WDYT @scikit-learn/core-devs ?

user: adrinjalali
issue_20582: state: open
title: Adapt centroids' initialisation in `BallTree`
body: As reported by @Witiko in https://github.com/scikit-learn/scikit-learn/discussions/20097#discussioncomment-743215, the centroids' initialization in `BallTree` is tight to the case of the euclidean distance regardless of the `DistanceMetric` set indirectly by the user provided `metric` argument.

https://github.com/scikit-learn/scikit-learn/blob/7b715111bff01e836fcd3413851381c6a1057ca4/sklearn/neighbors/_ball_tree.pyx#L71-L104 

The initialisation of those centroids should depends on the effective `DistanceMetric`. Apart from the euclidean case, are there closed form solutions for computing them?
user: jjerphan
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_20568: state: open
title: Initialize n_samples in classification criteria
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

N/A

#### What does this implement/fix? Explain your changes.

This PR correctly initializes the `n_samples` attribute of `ClassificationCriterion` class with the actual number of samples instead of a zero.

#### Any other comments?

This will enable use of criteria that require the total number of samples.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: breezerider
issue_20557: state: open
title: RegressorChain support for Pipelines including ColumnTransformer
body: #### Describe the bug

I can't seem to get the `RegressorChain` working with pipelines that include a `ColumnTransformer`. I posted an issue on StackOverflow with more: https://stackoverflow.com/questions/68430993/sklearn-using-regressorchain-with-columntransformer-in-pipelines .

Somewhere in `__init__.py / _get_column_indices(X, key)` this call fails: `all_columns = X.columns`  saying `'numpy.ndarray' object has no attribute 'columns'`. Because this is a known issue with `ColumnTransformer`, I suspect the `RegressorChain` can't be used with it.

I'm not sure if this is a supported scenario, but the documentation for RegressorChain (https://scikit-learn.org/stable/modules/generated/sklearn.multioutput.RegressorChain.html), for `set_params`, includes this:

"The method works on simple estimators **as well as on nested objects (such as Pipeline)**. The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object."

So I was led to assume it would also work with Pipelines including the column transformer.

#### Steps/Code to Reproduce

Any example with a Pipeline containing a ColumnTransformer and a Regressor. The StackOverflow link I included above has my code.

#### Expected Results

Fitted pipeline.

#### Actual Results

```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
C:\ProgramData\Anaconda3\envs\py38aml\lib\site-packages\sklearn\utils\__init__.py in _get_column_indices(X, key)
    373         try:
--> 374             all_columns = X.columns
    375         except AttributeError:

AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-181-24da1e03388c> in <module>
      3 
      4 chain_regressor = RegressorChain(base_estimator=chain_pipeline) #, order=[1,0,2])
----> 5 chain_regressor.fit(X, y)
      6 
      7 

C:\ProgramData\Anaconda3\envs\py38aml\lib\site-packages\sklearn\multioutput.py in fit(self, X, Y, **fit_params)
    840         self : object
    841         """
--> 842         super().fit(X, Y, **fit_params)
    843         return self
    844 

C:\ProgramData\Anaconda3\envs\py38aml\lib\site-packages\sklearn\multioutput.py in fit(self, X, Y, **fit_params)
    507         for chain_idx, estimator in enumerate(self.estimators_):
    508             y = Y[:, self.order_[chain_idx]]
--> 509             estimator.fit(X_aug[:, :(X.shape[1] + chain_idx)], y,
    510                           **fit_params)
    511             if self.cv is not None and chain_idx < len(self.estimators_) - 1:

C:\ProgramData\Anaconda3\envs\py38aml\lib\site-packages\sklearn\compose\_target.py in fit(self, X, y, **fit_params)
    205             self.regressor_ = clone(self.regressor)
    206 
--> 207         self.regressor_.fit(X, y_trans, **fit_params)
    208 
    209         return self

C:\ProgramData\Anaconda3\envs\py38aml\lib\site-packages\sklearn\pipeline.py in fit(self, X, y, **fit_params)
    339         """
    340         fit_params_steps = self._check_fit_params(**fit_params)
--> 341         Xt = self._fit(X, y, **fit_params_steps)
    342         with _print_elapsed_time('Pipeline',
    343                                  self._log_message(len(self.steps) - 1)):

C:\ProgramData\Anaconda3\envs\py38aml\lib\site-packages\sklearn\pipeline.py in _fit(self, X, y, **fit_params_steps)
    301                 cloned_transformer = clone(transformer)
    302             # Fit or load from cache the current transformer
--> 303             X, fitted_transformer = fit_transform_one_cached(
    304                 cloned_transformer, X, y, None,
    305                 message_clsname='Pipeline',

C:\ProgramData\Anaconda3\envs\py38aml\lib\site-packages\joblib\memory.py in __call__(self, *args, **kwargs)
    350 
    351     def __call__(self, *args, **kwargs):
--> 352         return self.func(*args, **kwargs)
    353 
    354     def call_and_shelve(self, *args, **kwargs):

C:\ProgramData\Anaconda3\envs\py38aml\lib\site-packages\sklearn\pipeline.py in _fit_transform_one(transformer, X, y, weight, message_clsname, message, **fit_params)
    752     with _print_elapsed_time(message_clsname, message):
    753         if hasattr(transformer, 'fit_transform'):
--> 754             res = transformer.fit_transform(X, y, **fit_params)
    755         else:
    756             res = transformer.fit(X, y, **fit_params).transform(X)

C:\ProgramData\Anaconda3\envs\py38aml\lib\site-packages\sklearn\compose\_column_transformer.py in fit_transform(self, X, y)
    503         self._validate_transformers()
    504         self._validate_column_callables(X)
--> 505         self._validate_remainder(X)
    506 
    507         result = self._fit_transform(X, y, _fit_transform_one)

C:\ProgramData\Anaconda3\envs\py38aml\lib\site-packages\sklearn\compose\_column_transformer.py in _validate_remainder(self, X)
    330         cols = []
    331         for columns in self._columns:
--> 332             cols.extend(_get_column_indices(X, columns))
    333 
    334         remaining_idx = sorted(set(range(self._n_features)) - set(cols))

C:\ProgramData\Anaconda3\envs\py38aml\lib\site-packages\sklearn\utils\__init__.py in _get_column_indices(X, key)
    374             all_columns = X.columns
    375         except AttributeError:
--> 376             raise ValueError("Specifying the columns using strings is only "
    377                              "supported for pandas DataFrames")
    378         if isinstance(key, str):

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

#### Versions

System:
    python: 3.8.10 (default, May 19 2021, 13:12:57) [MSC v.1916 64 bit (AMD64)]
executable: C:\ProgramData\Anaconda3\envs\py38aml\python.exe
   machine: Windows-10-10.0.22000-SP0

Python dependencies:
          pip: 21.1.3
   setuptools: 52.0.0.post20210125
      sklearn: 0.24.2
        numpy: 1.20.2
        scipy: 1.6.2
       Cython: None
       pandas: 1.2.5
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.2.0

Built with OpenMP: True
user: lokijota
labels: Hard: Hard level of difficulty
issue_20539: state: open
title: BLAS memory allocation error in KMeans & kNN & DBSCAN
body: #### Describe the bug
I have the following error:
```BLAS : Program is Terminated. Because you tried to allocate too many memory regions.```
When launch code:
```KMeans```
```python
from sklearn.datasets import make_blobs
from sklearn.cluster import KMeans

X, _ = make_blobs(n_samples=1000000, n_features=20, centers=1000)
model = KMeans(n_clusters=1000, max_iter=50, tol=0.0, n_init=1).fit(X)
model.predict(X)
```
```KNN```:
```python
from sklearn.datasets import make_classification
from sklearn.neighbors import KNeighborsClassifier
import pandas as pd
import numpy as np

X, y = make_classification(n_samples=20000, n_features=50)

model = KNeighborsClassifier(algorithm='brute', metric='euclidean', n_jobs=-1).fit(X, y)
model.predict(X)
```
```DBSCAN```:
```python
from sklearn.datasets import make_blobs
from sklearn.cluster import DBSCAN

X, _ = make_blobs(n_samples=500000, n_features=50, centers=100)

model = DBSCAN(eps=10., n_jobs=-1, min_samples=5, metric='euclidean', algorithm='auto').fit(X)
model.predict(X)
```

#### Steps/Code to Reproduce
Environment:
```
conda create -n bench -c conda-forge python=3.8
conda activate bench
pip install scikit-learn numpy pandas
```
Platform:
```
c5.24xlarge AWS EC2 Instance or c5a.24xlarge AWS EC2 Instance
Ubuntu 18.04.5 LTS (Bionic Beaver)
```
#### Expected Results
Nothing
#### Actual Results
```
BLAS : Program is Terminated. Because you tried to allocate too many memory regions.
```
#### Versions
```
scikit-learn == 0.24.2
numpy == 1.21.0
pandas == 1.3.0
scipy == 1.7.0
```
user: OnlyDeniko
issue_20537: state: open
title: Score decision tree using only samples that propagate to leaves trained with >N datapoints
body: After training a particular model, there is typically a decent amount of threshold tuning in order to select the appropriate decision rule for the application. In general, the goal is to identify rules (either based on threshold, or other properties of the algorithm) that enable us to exclude the model from making _bad_ decisions while still being applicable to the majority of the dataset.

For decision trees, one such possible rule is to exclude predictions which fall into leaves lacking a sufficient number of datapoints, where sufficient (>N) is specified by the user. For example, one might say that a leaf was added during training based on only 2 datapoints. During testing/scoring, we could skip samples which fall into this leaf to prevent low-confidence decision making.

Alternatives/nuance: it is possible that rather than raw number of samples, we may also want to consider impurity within the leaf, or perhaps some weighted metric weighing impurity against number of samples.

Feedback on whether this is something others have considered (I could not find any similar issues in the tracker) or whether there are clear drawbacks to this approach would be appreciated.

user: rohilverma
issue_20520: state: open
title: 'Shuffle' in the GroupKFold function and RepeatedGroupKFold function
body: Regarding #13619

### What is problem?
GroupKFold doesn't support 'shuffle' for repetition.  
That makes only deterministic combination like below. 

---
``` python
import sklearn
from sklearn.model_selection._split import GroupKFold
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([0, 0, 2, 2])
groups = np.array([0, 1, 2, 3])

group_kfold = GroupKFold(n_splits=2)
for train_index, test_index in group_kfold.split(X, y, groups):
    print("Train index:", train_index, " Test index:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    groups_train, groups_test = groups[train_index], groups[test_index]
    print("Train group:", groups_train, " Test group:", groups_test)
    print()
```
=== Output ===
Train index: [0 2]  Test index: [1 3]   
Train group: [0 2]  Test group: [1 3]  

Train index: [1 3]  Test index: [0 2]  
Train group: [1 3]  Test group: [0 2]  

---
We can combine group '[0 1], [2 3]' or '[0 3], [2 4]' also.
As we know, KFold functions already support 'shuffle' and 'random_state' argments for this issue.
But it wasn't implemented in the GroupKFold now.

### So, I suggest the feature in the GroupKFold and RepeatedGroupKFold like below.  

---
``` python
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([0, 0, 2, 2])
groups = np.array([0, 1, 2, 3])

group_kfold = GroupKFold(n_splits=2, shuffle=True, random_state=123456)
for train_index, test_index in group_kfold.split(X, y, groups):
    print("Train index:", train_index, " Test index:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    groups_train, groups_test = groups[train_index], groups[test_index]
    print("Train group:", groups_train, " Test group:", groups_test)
    print()
```
=== Output ===
Train index: [2 3]  Test index: [0 1]
Train group: [2 3]  Test group: [0 1]

Train index: [0 1]  Test index: [2 3]
Train group: [0 1]  Test group: [2 3]

``` python
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([0, 0, 2, 2])
groups = np.array([0, 1, 2, 3])

group_kfold = RepeatedGroupKFold(n_splits=2, n_repeats=2, random_state=2652124)
for train_index, test_index in group_kfold.split(X, y, groups):
    print("Train index:", train_index, " Test index:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    groups_train, groups_test = groups[train_index], groups[test_index]
    print("Train group:", groups_train, " Test group:", groups_test)
    print()
```
=== Output ===
Train index: [1 2]  Test index: [0 3]
Train group: [1 2]  Test group: [0 3]

Train index: [0 3]  Test index: [1 2]
Train group: [0 3]  Test group: [1 2]

Train index: [0 1]  Test index: [2 3]
Train group: [0 1]  Test group: [2 3]

Train index: [2 3]  Test index: [0 1]
Train group: [2 3]  Test group: [0 1]

---

I implemented this feature in a scikit-learn compatible way. (If not, please let me know)
Please find below source.

[Github source](https://github.com/BbChip0103/sklearn_repeated_group_k_fold/blob/main/sklearn_repeated_group_k_fold.py)
[Example in the colab](https://colab.research.google.com/drive/1OzRELL1vU15kEtA6OKTROz6ExekiJjL5?usp=sharing)

---

What do you think about this feature?
user: BbChip0103
issue_20494: state: open
title: Fix bug default marking of unlabeled data
body: 
#### Reference Issues/PRs

Fixes #17951 


#### What does this implement/fix? Explain your changes.

Implement `unlabeled_mark` as a parameter for BaseLabelPropagation.
This is a way for user to specify the way that they mark unlabeled data, 
which also fix the problem of described in issue #17951



#### Any other comments?

Thank you for reviewing this PR, if anything arise, please send me an email to
trantrikien239@gmail.com

user: trantrikien239
issue_20443: state: open
title: AdaBoost's training error can increase with a larger number of trees
body: We encountered the following behavior of AdaBoost when working on the [scikit-learn mooc](https://mooc-forums.inria.fr/moocsl/t/quiz-m6-03-question-3/3128/15). Here is a script that reproduces the problem:

We compute the validation curves for 3 models with different numbers of leaf nodes in the weak learners and each time study the impact of the number of trees on the train and validation errors.


```python
from sklearn.ensemble import AdaBoostRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import validation_curve
from sklearn.datasets import fetch_california_housing
import matplotlib.pyplot as plt


data, target = fetch_california_housing(return_X_y=True)

for max_leaf_nodes in [50, 100, 500]:
    adaboost = AdaBoostRegressor(
        learning_rate=1.,
        base_estimator=DecisionTreeRegressor(max_leaf_nodes=max_leaf_nodes),
    )
    param_range = [1, 2, 5, 10, 20, 50, 100, 500]
    train_scores, test_scores = validation_curve(
        adaboost,
        data,
        target,
        param_name="n_estimators",
        param_range=param_range,
        scoring="neg_mean_absolute_error",
        n_jobs=4,
    )
    train_errors, test_errors = -train_scores, -test_scores
    plt.figure()
    plt.errorbar(param_range, train_errors.mean(axis=1),
                 yerr=train_errors.std(axis=1), label="Training score",
                 alpha=0.7)
    plt.errorbar(param_range, test_errors.mean(axis=1),
                 yerr=test_errors.std(axis=1), label="Cross-validation score",
                 alpha=0.7)
    plt.legend()
    plt.xscale("log")
    plt.xticks(param_range)
    plt.ylabel("Mean absolute error in k$\n(smaller is better)")
    plt.xlabel("# estimators")
    _ = plt.title(
        f"AdaBoost regressor with {max_leaf_nodes=}\n"
        f"best CV score: {test_errors.mean(axis=1).min():.4f}"
    )
```

Here are the results:

![adaboost_50_leaves](https://user-images.githubusercontent.com/89061/124276287-e0752f00-db43-11eb-9a45-ae83a6507218.png)
![adaboost_100_leaves](https://user-images.githubusercontent.com/89061/124276290-e10dc580-db43-11eb-8848-32b8212019e6.png)
![adaboost_500_leaves](https://user-images.githubusercontent.com/89061/124276292-e1a65c00-db43-11eb-8525-8f8da999477e.png)

For 500 leaves we get the expected behavior: both the training and validation errors decrease when increasing the number of trees. At some point we reach a plateau of diminishing returns (expected, at least for validation error).

However for lower number of leaves (e.g. the second plot with `max_leaf_nodes=50`), the training set is growing with more trees! This is really something that I would not have expected. I wonder if this is not revealing a bug in our implementation.

For reference, for lower values of `max_leaf_nodes` both the training and validation errors stay do not decrease (and even increase again) and they stay close to one another (severe under fitting) with large error bars.
user: ogrisel
issue_20341: state: open
title: Confusion_matrix: drilldown to underlying data.
body: I'm creating a confusion_matrix for the oxford102 UK flower dataset with the following code:

```
    class_names = {"21": "fire lily", "3": "canterbury bells", "45": "bolero deep blue", ... }
    class_names_sorted=sorted(class_names.items(), key=lambda x: int(x[0]))

    ....

    history = model.fit(train_batches, epochs=args.epochs,
              validation_data=val_batches,
              callbacks=callbacks)

    y_pred = np.argmax(model.predict(test_batches), axis=-1)
    l1=[]
    for i in ds_test: l1.append(i['label'].numpy())
    score=accuracy_score(l1, y_pred)
    print("accuracy_score against test_batches = "+str(score))

    # declare global so I can still see the variable after program has exited.
    global cm
    cm = confusion_matrix(l1, y_pred, labels=np.arange(1,103))
    fig, ax = plot_confusion_matrix(conf_mat=cm, figsize=(30,30), class_names = class_names_sorted, show_normed=False)
    plt.tight_layout()
    dateAndTime3 = datetime.now().strftime("%Y-%m-%d-%H%M%S")
    fig.savefig("cm-sorted-"+str(dateAndTime3)+".png")
```

This produces a 102X102 cm that's unwieldy but not unuseful.

The problem is that this is the first time I've used a confusion_matrix and while I've stepped through some of the sklearn (and mlxtend) code I'm unsure of the mapping from the cells down onto the underlying data.  That is, can I rely on the buckets to correspond correctly to the underlying data (species of flowers)?

What would be useful is some sort of drilldown - from the graphic - to the data so that I can confirm that I'm looking at what I think I'm looking at.

Since I'm new to this codebase I don't have a proposed solution.
user: patfla
issue_20337: state: open
title: Numerical stability problems in GaussianProcessRegressor on 32 bit Python
body: See the failing tests observed on 32 bit Linux with Python 3.7 or 3.8 in #20334 for details.

Those tests are skipped for now to avoid blocking the nightly build and release processes.
user: ogrisel
issue_20335: state: open
title: Test failure: test_loadings_converges in test_pls.py
body: Observed on travis last night, possibly caused by the upgrade of numpy / scipy:

```python
___________________________ test_loadings_converges ____________________________
[gw1] linux -- Python 3.7.10 /tmp/tmp.J9NDQpQnRJ/venv/bin/python

    def test_loadings_converges():
        """Test that CCA converges. Non-regression test for #19549."""
        X, y = make_regression(n_samples=200, n_features=20, n_targets=20, random_state=20)
        cca = CCA(n_components=10, max_iter=500)
        with pytest.warns(None) as record:
            cca.fit(X, y)
        # ConvergenceWarning is not raised
>       assert not record
E       assert not WarningsChecker(record=True)
/tmp/tmp.J9NDQpQnRJ/venv/lib/python3.7/site-packages/sklearn/cross_decomposition/tests/test_pls.py:583: AssertionError
```

I haven't investigated the cause yet. I can probably give it a look interactively in a linux arm64 docker container on my M1 laptop.
user: ogrisel
issue_20319: state: open
title: [RFC] critical difference diagrams for multiple model performance comparison
body: I have the impression that more and more machine learning papers use "critical difference diagrams" to summarize analysis of the predictive performance of a group of models.

In particular this method is implemented in https://github.com/hfawaz/cd-diagram under the GPLv3 license (with dependencies on pandas and networkx).

I wonder if we should not include such a plotting utility directly in scikit-learn. However the license and the dependencies are not compatible with scikit-learn.

Furthermore I cannot google a reference that explains the "Wilcoxon-Holm post-hoc analysis".

If we find a way to have a scikit-learn compatible implementation (without extra deps beyond numpy and matplotlib) I think this would be a very nice extension to @martinagvilas' tutorial on statistical analysis of model performance:

https://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_stats.html

- Would @hfawaz be ok if scikit-learn decides to ship a reimplementation (with proper credits) under the BSD license terms?
- Would it be complex to write such an implementation without the networkx dependency?
- If this is too complex, would scikit-learn be ok to have networkx as an uptional dependency just for this utility?
- Are there any known drawbacks in the use of this tool to communicate relative model performance conclusions?

I am interested in anybody's opinion on this potential feature request.
user: ogrisel
issue_20287: state: open
title: FIX ColumnTransformer raise TypeError when remainder columns have incompatible dtype
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
This is a fix to #20090 

#### What does this implement/fix? Explain your changes.
As discussed in the issue thread, the error is raised because the remainder columns that got passed through has incompatible dtype. This fix converts all the columns into a common dtype explicitly before calling `np.hstack` to avoid the error.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: MaxwellLZH
labels: Needs Decision: Requires decision
issue_20266: state: open
title: Precision @ Recall K || Recall @ Precision K
body: #### Describe the workflow you want to enable

Enable users to conveniently calculate "Precision at Recall K" or "Recall at Precision K" on the probabilities from clf.predict_proba.  

```
clf.fit(X_train, y_train)
y_pred = clf.predict_proba(X_test)


p_at_r99 = precision_at_recall_k(y_test, y_pred, k=.99)
```

#### Describe your proposed solution

`precision_at_recall_k` (also it's complement `recall_at_precision_k`) 

#### Describe alternatives you've considered, if relevant

I've written this code myself probably 10 separate times. Facebook uses this metric pretty heavily, and they have coded up this functionality here: https://github.com/facebookresearch/mmf/blob/322bd33d4a67f5bbe806624d9afd3c2cf921ec50/mmf/modules/metrics.py#L1020 

#### Additional context
user: Ryanglambert
issue_20233: state: open
title: [MRG] FIX "Add the ability for the user to provide specific landmarks points to approximate the Nystroem kernel approximation "
body: Dear Developers,

This PR Fixes #20232.

The `fit `method of the `Nystroem `class is refined so that it is possible to pass landmarks points. Also with the update, the documentation has been changed and tests have been added. 
user: hurricane642
issue_20232: state: open
title: Add the ability for the user to provide specific landmarks points to approximate the Nystroem kernel approximation
body: Dear developers, 
I would to ask for support for the provide specific landmarks points to approximate the Nystroem kernel approximation. In the current version of the code, the points are chosen randomly. In this case it is often more convenient for the user to select them beforehand using some method and already build the approximation using this method. 

Possible solution
To be presented in a separate PR

user: hurricane642
issue_20226: state: open
title: Elastic Net Regression with Linear Constraints
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->
Dear Developers,
I would to ask for support for the elastic net regression with linear constraints, i.e.
`minimize 1 / (2 * n_samples) * ||y - Xw||^2_2 + alpha * l1_ratio * ||w||_1 + 0.5 * alpha * (1 - l1_ratio) * ||w||^2_2`
subject to:
`Aw=b; Cw<=d`
This kind of problems are quite common and usually are needed to impose shape constraints on the fitting function.

#### Describe your proposed solution
The problem has been addressed in several papers, e.g.

- [Gaines, Brian R., Juhyun Kim, and Hua Zhou. "Algorithms for fitting the constrained lasso." Journal of Computational and Graphical Statistics 27.4 (2018): 861-871.](https://hua-zhou.github.io/media/pdf/GainesKimZhou08CLasso.pdf)

- [James, Gareth M., Courtney Paulson, and Paat Rusmevichientong. "The constrained lasso." Refereed Conference Proceedings. Vol. 31. 2012.](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.306.8795&rep=rep1&type=pdf)

I'm not sure yet which is the most efficient approach to solve those problems.

user: gmosem
issue_20224: state: open
title: AttributError when accessing `coef_` after constructor for `ElasticNet` subclasses
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->
#### Describe the bug
Accessing attribute `coef_` after object constructor results in an `AttributeError`.

Context: Using duck typing for scikit-learn models to test if the attribute `coef_` or `feature_importance_` is present on the object and, if not, abort early without having to call `fit()` first.

#### Steps/Code to Reproduce
Using 
```
from sklearn.linear_model import Lasso
model = Lasso()
model.coef_
```
results in
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Lasso' object has no attribute 'coef_'
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
To be discussed. Options are
* Specific exception, e.g., `NotFittedError` is raised
* None is returned

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
AttributeError is raised.

#### Versions
System:
    python: 3.7.9 (default, Feb  9 2021, 09:08:29)  [GCC 6.3.0 20170516]
executable: /usr/local/bin/python
   machine: Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-debian-9.13

Python dependencies:
          pip: 21.0.1
   setuptools: 56.1.0
      sklearn: 0.24.2
        numpy: 1.20.3
        scipy: 1.6.3
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True

<!-- Thanks for contributing! -->

user: pfaeffli
issue_20217: state: open
title: Some estimator checks on meta-estimators are not invalid
body: Now that we are trying to move meta-estimators to delegate validation to inner estimators, I think that some estimator checks are invalid for meta-estimators. For example, `check_complex_data` on a meta-estimator that delegates validation, actually checks that the inner estimator is validating. Should we skip the input validation checks for meta-estimators?

CC @ogrisel 

XREF https://github.com/scikit-learn/scikit-learn/pull/20205
user: thomasjpfan
labels: module:test-suite: everything related to our tests
issue_20211: state: open
title: Word n-grams for Devanagari script are not generated correctly.
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
<!---->
Word n-grams for Devanagari script are not generated correctly. Tokens are formed only until a modifier (maatra) is encountered, which should not be the case. It should be until you find a whitespace or a punctuation mark.

#### Steps/Code to Reproduce
<!--
Please add a minimal example that we can reproduce the error by running the
code. Be as succinct as possible, do not depend on external data. In short, we
are going to copy-paste your code and we expect to get the same
result as you.
-->
Example:
```python
d = ['आई- जाइयै नुहाड़ा ध्यान उस्सै भेठा जा’रदा हा।',
     'य बात सुणते ही सब गौं वाल नजदीक जंगलै तरफ भाज् और बांनर पकड़ पकड़ बेर 100- 100 रु में बेचंण लाग्।']

v = CountVectorizer()
x = v.fit_transform(d)
f = v.get_feature_names()
print(f)
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
`['100', 'आई', 'उस्सै', 'और', 'गौं', 'जंगलै', 'जा', 'जाइयै', 'तरफ', 'ध्यान', 'नजदीक', 'नुहाड़ा', 'पकड़', 'बांनर', 'बात', 'बेचंण', 'बेर', 'भाज्', 'भेठा', 'में', 'य', 'रदा', 'रु', 'लाग्', 'वाल', 'सब', 'सुणते', 'हा', 'ही']`

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
`['100', 'आई', 'इय', 'उस', 'और', 'गल', 'णत', 'तरफ', 'नजद', 'नर', 'पकड', 'रद', 'सब']`

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
import imblearn; print("Imbalanced-Learn", imblearn.__version__)
-->

```
System:
    python: 3.7.10 (default, May  3 2021, 02:48:31)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.109+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 19.3.1
   setuptools: 57.0.0
      sklearn: 0.24.2
        numpy: 1.19.5
        scipy: 1.4.1
       Cython: 0.29.23
       pandas: 1.1.5
   matplotlib: 3.2.2
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```

<!-- Thanks for contributing! -->

user: rachanagusain
issue_20178: state: open
title: Optional `stop_words_` attribute for CountVectorizer and TfidfVectorizer
body: #### Describe the workflow you want to enable
Hi,

Refering to this message found for `sklearn.feature_extraction.text.CountVectorizer` and `sklearn.feature_extraction.text.TfidfVectorizer`: 
```
The stop_words_ attribute can get large and increase the model size when pickling. This attribute is provided only for introspection and can be safely removed using delattr or set to None before pickling.
```
This attribute can indeed be very large when the dataset is big (leading to out of memory error sometimes). Besides, what it contains is not used by everyone (not always).
So, it would be nice to have this attribute as optional. 
#### Describe your proposed solution
* Add the possibility to indicate that we don't want to build this attribute when initializing `CountVectorizer` and `TfidfVectorizer` 

#### Describe alternatives you've considered, if relevant

#### Additional context

user: xiangb
issue_20167: state: open
title: Enable mixed ensembles with estimators that do & don't accept the sample_weight fit_param
body: I need to make a VotingRegressor ensemble with some estimators that accept sample weights during fitting and some that don't.  Currently, mixed ensembles raise an exception:
```py
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor, VotingRegressor
from sklearn.neighbors import KNeighborsRegressor

X, y = make_regression()
weights = abs(y)

rgr = VotingRegressor(estimators=[('LR', LinearRegression()), 
                                  ('KNN', KNeighborsRegressor()),
                                  ('XGBR', RandomForestRegressor())])

rgr.fit(X, y, sample_weight=weights)
```
**result:**
TypeError: Underlying estimator KNeighborsRegressor does not support sample weights.

A possible solution would be to have the ensemble class (e.g. VotingRegressor, VotingClassifier, StackingRegressor, StackingClassifier) read the `fit()` signatures of the estimators in the ensemble and not pass sample_weight to estimators that don't accept sample_weight.  Or more realistically, catch exceptions caused by calls to `fit()` with the sample_weight parameter and then default to calling `fit()` without this parameter.  This behavior could be default, or activated by flag like "enable_mixed_sample_weight" in the ensemble class's `__init__` method.  If it's important to notify the user when an estimator doesn't accept the sample_weight parameter, notification and the exception currently in place could be enabled with a flag like "enforce_sample_weight".

As a workaround I'm using the `Ensemble` class from the pipecaster library (https://github.com/ajcallegari/pipecaster) which allows mixed ensembles by catching exceptions caused by `fit()` and then defaulting to a `fit()` call without the sample_weight parameter.  This `Ensemble` class has the scikit-learn interface and supports classification, regression, voting, and model stacking.
user: ajcallegari
issue_20162: state: open
title: Metrics for prediction intervals
body: Yesterday, I had a dream where scikit learn has implemented main metrics for prediction intervals.

A review of typical metrics for prediction intervals can be found on this publication (p.16 to 17): 'Review of Deterministic and Probabilistic Wind Power Forecasting: Models, Methods, and Future Research' (https://www.mdpi.com/2673-4826/2/1/2)

Typically : with <img src="https://render.githubusercontent.com/render/math?math=L_{t}"> the lower bound and <img src="https://render.githubusercontent.com/render/math?math=U_{t}"> the upper bound


- PICP (Prediction Interval Coverage Probability)
![image](https://user-images.githubusercontent.com/11358792/122648808-17eae100-d12b-11eb-8fd2-788f0161a507.png)
with 
![image](https://user-images.githubusercontent.com/11358792/122648832-381aa000-d12b-11eb-894d-6ae9ead9019c.png)
PICP is maybe not so useful in itself, but needed to calculate the ACE below, which is really the critical metric.

- PINC (Prediction Interval Nominal Coverage) : the nominal level of uncertainty, e.g. 90% if the quantiles you predict are 5% and 95%
PINC is somewhat a necessary preliminary definition for computing the ACE below.

- ACE (Average Coverage Error) 
![image](https://user-images.githubusercontent.com/11358792/122648875-65674e00-d12b-11eb-912c-d7b5fb4940ea.png)
That metric is really the critical one that really feels missing the most. It represents how much the interval can be trusted, because, for example, when we say that we compute the 90 % confidence interval, we want that close to 90% of the test points rely in that interval. For this reason we usually refer to that metric as measuring the _reliability_ of the interval.

- PINAW (Prediction Interval Normalized Average Width)
![image](https://user-images.githubusercontent.com/11358792/122649001-0eae4400-d12c-11eb-9861-057551ff8fbf.png)
That is the second critical metric that would need be implemented. It measures the _sharpness_ of the interval. This is a very complementary thing to the ACE, as it measures how wide the interval is. Indeed, useful intervals are the one that are reliable of course, but that are also narrow. The ACE does not mesure how narrow/sharp we are, this needs to be measured by the PINAW (or interval score, see below)

- CWC (Coverage Width Criterion)
![image](https://user-images.githubusercontent.com/11358792/122648945-b6774200-d12b-11eb-8fcc-4bc659f25a2f.png)
with 
![image](https://user-images.githubusercontent.com/11358792/122648962-c4c55e00-d12b-11eb-9ad4-66c4e8c35830.png)
<img src="https://render.githubusercontent.com/render/math?math=\mu"> usually equals the PINC and <img src="https://render.githubusercontent.com/render/math?math=\eta"> is a penalty coefficient
That metric I have put here for reference because it is cited in the article. For the moment I personnaly have not used it as it combines the ACE and PINAW, but in a way that still needs to be clarified (to me at least, before I would chose to use it).

- Interval Score (This metric does not come from the same publication as the others, this one comes from Wan, C., Xu, Z., Pinson, P., Dong, Z. Y., & Wong, K. P. (2014). Probabilistic forecasting of wind power generation using extreme learning machine. IEEE Transactions on Power Systems, 29(3). https://doi.org/10.1109/TPWRS.2013.2287871)
![image](https://user-images.githubusercontent.com/11358792/142428632-2d5e560d-7184-40c3-9c6d-46b64e3df227.png)
With 
![image](https://user-images.githubusercontent.com/11358792/142436825-38784be5-312b-4e9a-afe4-c69a8d579888.png) defining the width of the PI.
That is probably the third metric I would find very interesting to add. The reason for this is that the interval score adds to the ACE and PINAW the consideration of how far is the prediction from the interval when it is not in the interval, which can permit to discriminate between two models where predictions would be out of the interval, but the prediction of one model would be closer to the interval's boundary than the prediction of the other model.

Currently sklearn permits to calculate such prediction intervals, typically with GradientBoostingRegressor and quantile loss, but the classic way to calculate the validity of these intervals cannot be done through sklearn as smoothly as for deterministic forecast with MSE, MAE, etc. Implementing several of these metrics would close the gap.
user: ThomasBourgeois
issue_20144: state: open
title: TST suppress convergence warnings in coordinate descent
body: #### Reference Issues/PRs

closes #4780

#### What does this implement/fix? Explain your changes.

These warnings have been fixed by either increasing the number of
iterations of fitting, or by suppressing the warnings with pytest.warns

For parametrized tests a check is done so that only the failing tests
get the pytest.warns context manager, all other get a dummy context
manager that doesn't do anything.

#### Any other comments?

There is one instance where the warning does not disappear in
   test_linear_model_sample_weights_normalize_in_pipeline
(second context manager call), but I don't know why ...


user: norbusan
issue_20132: state: open
title: Follow-up of QuantileRegressor implementation
body: `QuantileRegressor` has been implemented in https://github.com/scikit-learn/scikit-learn/pull/9978
However, it was reported to have an issue with large array of data where linear programming solvers require to form huge matrices for the optimization problem.

A potential solution is to have an online solver using `partial_fit`. However, it is not clear which solvers would be best to solve this issue.

Has proposed by @agramfort https://github.com/scikit-learn/scikit-learn/pull/9978#issuecomment-847159716, we should benchmark the different available solvers before making an implementation choice.

Pinging @RPyElec @avidale @agramfort @lorentzenchr @GaelVaroquaux that were involved in the discussion in the former pull-request.

**Tasks:**

- [x] Add support for sparse `X` (relatively easy). #21086
- [ ] Propose an algorithm for a `partial_fit` method, test and benchmark it (difficult task).
user: glemaitre
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_20090: state: open
title: ColumnTransformer give "TypeError: invalid type promotion"
body: #### Versions
sklearn 0.23.2

####  Description
Instantiating `ColumnTransformer` with the remainder argument set to "passthrough" produces a TypeError under certain circumstances. I narrowed down one such circumstance. 

The error occurs when exactly _n - 1_ columns are transformed (where _n_ is the total number of columns) and the one column that gets passed through (i.e., not transformed) has a dtype that cannot be converted to that of the other columns. The root cause is that sklearn tries to combine the arrays with `numpy.hstack` and fails.

#### Code to Reproduce
````python
from sklearn import preprocessing, compose
import numpy as np
import pandas as pd
import datetime

prng = np.random.default_rng()
d = pd.DataFrame(prng.random((20,3)), columns = ["aaa", "bbb", "ccc"])
d["time"] = datetime.datetime.now()

columns = ["aaa", "bbb", "ccc"]
t = compose.ColumnTransformer(
    [("stnd", preprocessing.StandardScaler(), columns)],
    remainder="passthrough"
)
t.fit_transform(d)
````
The above code produces "TypeError: invalid type promotion". The dtype of the "time" column is datetime and that of the others is float. Like described above, letting only the "time" column to pass through results in hstack failing when it tries to concatenate the arrays. If you change columns to `columns = ["aaa", "bbb"]`, it works as is expected. Also changing the remainder argument to `remainder="drop"` also works.
user: princyok
issue_20074: state: open
title: Opening the mutual information computation API for arbitrary variable sets
body: Mutual information regression has been implemented in `sklearn.feature_selection` and can compute mutual information between the main kinds of variables (discrete/discrete, continuous/discrete, continuous/continuous). However, the present API only to compute only the mutual information between some input variables and a target.

There is also an implementation of MI score in the `sklearn.metrics` module which is dedicated to discrete / discrete variables and is poroposed to assess the quality of unsupervized clustering mainly.

It would be nice to have a more convenient API allowing to compute MI between any pairs of variables in order to obtain a matrix of pairwise MI scores.

Some normalized versions of the MI values should also be considered. 

user: AlexisMignon
issue_20070: state: open
title: FIX use correct formulation for ExpSineSquared kernel
body: closes #19343
closes #9824

TODO:

- [ ] Add reference to the original paper: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.81.1927&rep=rep1&type=pdf
- [ ] Add test to check kernel and kernel gradients values
- [ ] It will have an impact on some example that uses this kernel outside of the gaussian process.
user: glemaitre
issue_20064: state: open
title: Adding user-provided validation data added to randomforest models other than using non selected indices from bootstrapping
body: Although using non selected indices from bootstrapping is a very nice implementation without cross validation, in cross validation it leads  to possible validating on same indices multiple times. It might be a question answered  to why not using it but I couldn't find any discussion.
user: siavrez
issue_20059: state: open
title: Unbiased mean decrease in impurity if tree-based methods
body: This issue is a follow-up of the PR #20058

## Background

We are aware that our current implementation of mean decrease in impurity is biased:

- it uses statistic from the training set (issue with models that overfit)
- it favours continuous feature and large cardinality categorical features

## Current solution

To overcome the issue, we proposed the following:

- implementation of the feature permutation importance. It tends to be computationally expensive but it has the advantage to be model agnostic.
- implementation of the feature permutation importance leveraging the out-of-bag samples available in the random forests: https://github.com/scikit-learn/scikit-learn/pull/18603. This method is limited to random forest (it requires out-of-bag samples).

## Public API

Currently, the biased feature importance is available via the fitted attribute `feature_importances_` in the tree-based model (decision tree, random forest, gradient-boosting decision tree).

#18603 would introduce a parameter to the random forests, `feature_importances` to be set to `{"permutation_oob", "impurity"}`, to switch between mean decrease in impurity to out-of-bag samples feature permutation importance.

## Correct the biased of mean decrease in impurity

@ZhengzeZhou proposed an alternative to the above implementation in #20058. In short, it leverages the out-of-bag samples to correct the bias of the mean decrease in impurity.

While reviewing the paper, I came across recent work tackling the same issue using the out-of-bag samples:

- https://arxiv.org/pdf/1903.05179.pdf implemented in #20058 
- https://arxiv.org/pdf/1906.10845.pdf
- https://arxiv.org/pdf/2003.02106.pdf

@ZhengzeZhou I could not yet look at all papers in-depth but you probably went across them since they are tackling the same issue as your research. Do you have any feedback regarding the proposals?

Also, all methods are quite recent and while none of the proposals are fulfilling the inclusion criterion, we might consider an implementation to solve this issue since this is problematic. However, it should be discussed in this issue before starting any pull request.

## API discussion

Unbiasing the mean decrease in impurity is requiring out-of-bag samples. Currently, only random forests can benefit from this feature.

While I worked on #18603, it seems that adding a parameter to select the feature importance was the best. In the future, we could think about deprecating the MDI computation on the training set.

However, we still have issues when we don't have out-of-bag samples at hand. I am unsure about what would be the best options for decision tree and gradient boosting decision tree classes.

user: glemaitre
labels: Needs Decision - API: Requires decision regarding API
issue_20051: state: open
title: sklearn.tree.export_dict
body: Currently, there are two options to get the decision tree representations: `export_graphviz` and `export_text`.

I would like to add `export_dict`, which will output the decision as a nested dictionary.

While you can convert the graphviz representation using cli tools, it's a bit unruly and is a weird workflow. When converting back, the nesting is also kind of weird, with `/n` chars floating around. 

The main utility for json/dict representations are

1. That they're far easier to work with for any front-end work. 
2. Working with a dict allows you to use the representation easily for any downstream tasks using the decision tree (that aren't raw inference).



user: jwilber
issue_20035: state: open
title: Allow stacking pandas dataframes in ColumnTransformer?
body: Right now we make it easy to overwrite ``_hstack`` in ColumnTransformer, but I wonder if we should try to be more generic in the first place. I don't think without something like nep 37 we can do fully generic stacking, but maybe we can already allow stacking pandas dataframes?

I can think of few cases where all the transformers output pandas dataframes but you want to output a numpy array in the ColumnTransformer.
This would be a backward incompatible change, so we'd probably need to make it optional (or deprecate the current behavior).
Wdyt?
user: amueller
issue_20034: state: open
title: CategoricalNB : it needs to add proper treatment  for unseen values in validation data  (validation data has new values than train data )
body: 

#### Describe the workflow you want to enable

CategoricalNB : it needs to add proper treatment  for unseen values in validation data  (validation data has new values than train data )
to
https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.CategoricalNB.html

meantime OrdinalEncoder put all unseen to one category - 0
CategoricalNB assumes that the sample matrix  is encoded (for instance with the help of OrdinalEncoder) such that all categories for each feature  are represented with numbers 
 where 
 is the number of available categories of feature .

using min_categories
also is not solving  this problem 

#### Describe your proposed solution
not sure
but unseen values in data row should be used for inference 



user: Sandy4321
issue_20033: state: open
title: enable regression step coefs after fit
body: For a lot of different analyses that come upon model training, in special for PLSRegression and the like, it is necessary to have the resulting coefficients of the regression step only, the "alpha" coefficients in the documentation 1.8.1.2. Predicting the targets Y
https://scikit-learn.org/stable/modules/cross_decomposition.html#cross-decomposition

I suggest including them in future releases of sklearn
user: vfonsecad
issue_20032: state: open
title: LatentDirichletAllocation uses inconsistent normalization for test perplexity in the components_ variable
body: #### Describe the bug
In the calculation of the perplexity in beta variable, the normalizer uses [the components_ variable from the training distribution](https://github.com/scikit-learn/scikit-learn/blob/a9ce392f3a58da5caf5ac9bd287205e220082fc5/sklearn/decomposition/_lda.py#L737), whereas the denominator uses the testing word counts similar to the other perplexity terms. This seems inconsistent. We should change the denominator to the training word counts for consistency.

Another reason to propose this change is that, conventionally, the beta variable should be fixed at its MAP value. This is possible by feeding the training data multiple times to sharpen the beta distribution. With the proposed fix, the entropy contribution from the beta components would converge to zero as the training sample size increases to infinity.

#### Steps/Code to Reproduce
Need to add a few printers in the source code. Please reference 
https://github.com/yifeim/scikit-learn/tree/bugfix_lda

Then run sklearn example modified from Olivier Grisel, Lars Buitinck, Chyi-Kwei Yau:
```python
from time import time
import matplotlib.pyplot as plt

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.decomposition import NMF, LatentDirichletAllocation
from sklearn.datasets import fetch_20newsgroups

n_samples = 2000
n_features = 1000
n_components = 10
n_top_words = 20

# Load the 20 newsgroups dataset and vectorize it. We use a few heuristics
# to filter out useless terms early on: the posts are stripped of headers,
# footers and quoted replies, and common English words, words occurring in
# only one document or in at least 95% of the documents are removed.

print("Loading dataset...")
t0 = time()
data, _ = fetch_20newsgroups(shuffle=True, random_state=1,
                             remove=('headers', 'footers', 'quotes'),
                             return_X_y=True)
data_samples = data[:n_samples]
print("done in %0.3fs." % (time() - t0))

# Use tf (raw term count) features for LDA.
print("Extracting tf features for LDA...")
tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2,
                                max_features=n_features,
                                stop_words='english')
t0 = time()
tf_vectorizer.fit(data)
tf = tf_vectorizer.transform(data_samples)
tt = tf_vectorizer.transform(data[n_samples:2*n_samples])
print("done in %0.3fs." % (time() - t0))
print()

print('\n' * 2, "Fitting LDA models with tf features, "
      "n_samples=%d and n_features=%d..."
      % (n_samples, n_features))
lda = LatentDirichletAllocation(n_components=n_components, max_iter=5,
                                learning_method='online',
                                learning_offset=50.,
                                random_state=0)
t0 = time()
lda.fit(tf)
print("done in %0.3fs." % (time() - t0))

print('\n\nPrinting perplexity with details')
lda.verbose=1
print(f"scikit-learn training perplexity {lda.perplexity(tf):.3f}")
print(f"scikit-learn testing perplexity {lda.perplexity(tt):.3f}")
```

#### Expected Results
Test perplexity should be around 1461 after the proposed fix.

#### Actual Results
entropy phi: 6.118 theta: 0.146 beta: original 0.172 proposed fix 0.174
scikit-learn training perplexity 623.519
entropy phi: 6.986 theta: 0.127 beta: original 0.139 proposed fix 0.174
scikit-learn testing perplexity 1411.090

#### Versions
https://github.com/yifeim/scikit-learn/tree/bugfix_lda

based on the following version:
```
System:
    python: 3.6.13 | packaged by conda-forge | (default, Feb 19 2021, 05:36:01)  [GCC 9.3.0]
executable: /home/ec2-user/anaconda3/envs/pytorch_p36/bin/python
   machine: Linux-4.14.225-121.357.amzn1.x86_64-x86_64-with-glibc2.9

Python dependencies:
          pip: 21.0.1
   setuptools: 49.6.0.post20210108
      sklearn: 1.0.dev0
        numpy: 1.19.5
        scipy: 1.5.3
       Cython: 0.29.22
       pandas: 1.1.5
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```
user: yifeim
issue_20029: state: open
title: Graph Spectral Embedding to sklearn/manifold
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes #18177, which was also further discussed [here](https://github.com/scipy/scipy/issues/13486)

#### What does this implement/fix? Explain your changes.
Adds new class `GraphSpectralEmbedding` to sklearn/manifold for spectral embedding of graph represented data, effective on directed and undirected graphs.
The two methods added specifically are Adjacency Spectral Embedding (ASE), and Laplacian Spectral Embedding (LSE).

#### Any other comments?
Our current implementations in [microsoft/graspologic,](https://microsoft.github.io/graspologic/tutorials/embedding/AdjacencySpectralEmbed.html) (available at [ASE](https://github.com/microsoft/graspologic/blob/dev/graspologic/embed/ase.py) and [LSE)](https://github.com/microsoft/graspologic/blob/dev/graspologic/embed/lse.py) input a graph represented as a dense or sparse array, and return the appropriate embedding.

user: asaadeldin11
issue_20006: state: open
title: StandardScaler with_mean=False fits mean_
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
<!--
A clear and concise description of what the bug is.
-->

Hi,

The optional parameter `with_mean` of the StandardScaler is not respected in all cases.

Notably, for simply normalizing a column vector, it is ignored and the mean is actually computed when either `fit` or `fit_transform` is called.

#### Steps/Code to Reproduce

Pretty self-explanatory:

```python
from sklearn.preprocessing import StandardScaler

x = np.array([0, 0, 1, 1]).reshape(-1, 1)

scaler = StandardScaler()
print(getattr(scaler, "mean_", None), getattr(scaler, "scale_", None)) # >>> None None
scaler.fit(x)
print(getattr(scaler, "mean_", None), getattr(scaler, "scale_", None)) # >>> [0.5] [0.5]

scaler = StandardScaler(with_mean=False)
print(getattr(scaler, "mean_", None), getattr(scaler, "scale_", None)) # >>> None None
scaler.fit(x)
print(getattr(scaler, "mean_", None), getattr(scaler, "scale_", None)) # >>> [0.5] [0.5]
```

Looking at the source code, in the case starting line 827 of `scikit-learn/_data.py` there is clearly no case that handles the case where only `with_mean` is set to `False`. 



#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
import imblearn; print("Imbalanced-Learn", imblearn.__version__)
-->

```
System:
    python: 3.7.9 (default, Apr 20 2021, 15:00:03)  [Clang 12.0.0 (clang-1200.0.32.29)]
executable: /Users/.../.venv/bin/python
   machine: Darwin-20.3.0-x86_64-i386-64bit

Python dependencies:
          pip: 21.0.1
   setuptools: 54.2.0
      sklearn: 0.24.2
        numpy: 1.20.2
        scipy: 1.6.3
       Cython: None
       pandas: 1.0.5
   matplotlib: 3.4.1
       joblib: 1.0.1
threadpoolctl: 2.1.0
```


<!-- Thanks for contributing! -->

user: pierresegonne
labels: Needs Decision - API: Requires decision regarding API
Needs Decision - Backward Compatibility: Requires decision regarding backward compatibility
issue_19995: state: open
title: Revisit GaussianProcessRegressor for multioutput
body: While working on https://github.com/scikit-learn/scikit-learn/pull/19952, I needed quite some time to realize that the formulation in the original reference (http://www.gaussianprocess.org/gpml/chapters/RW.pdf) is not natively handling multioutput.

While it seems possible, we have very little information regarding this case. Indeed, there are several ways to handle the multi-output cases that I don't know anything about :)

However, I think that we should make sure that we are not supporting this use case badly. I think that we could have a look to https://papers.nips.cc/paper/2007/file/66368270ffd51418ec58bd793f2d9b1b-Paper.pdf
user: glemaitre
issue_19973: state: open
title: Fix Common tests xfailed for Bicluster estimators issue #19548
body: #### Reference Issues/PRs
Fixes #19548 

#### Implementation Specification

Fixed check_fit2d_1sample by not calling the squeeze method on a 1 dimensional array in  "sklearn\cluster\_bicluster.py::_scale_normalize(X)"

Fixed check_estimator_sparse_data by setting all nan and inf values to 0 in "sklearn\cluster\_bicluster.py::SpectralCoclustering._fit(self, X)"

Fixed check_estimators_dtypes by setting all nan and inf values to a num value and fixing a typo (X_scaled -> X_scaled.data) in "sklearn\cluster\_bicluster.py::_bistochastic_normalize(X, max_iter=1000, tol=1e-5)"

check_fit2d_1feature,
check_methods_subset_invariance,
check_dont_overwrite_parameters, 
and check_fit2d_predict1d
all fixed by the former resolutions as well



user: AmyLExtraordinaire
issue_19972: state: open
title: AdaBoost (and others?) should allow not to normalize feature_importances_ at the learner level (and do so by default?)
body: #### Describe the bug

`AdaBoostClassifier.feature_importances_` makes a weighted average of importances...

https://github.com/scikit-learn/scikit-learn/blob/9b7ff272534f130893e95933db46a3ff295190b2/sklearn/ensemble/_weight_boosting.py#L256

... which however are _already normalized at the weak learner level_.

This makes the comparison of importance (_across_ weak learners) potentially meaningless. In particular, in the simplest possible case of one-rule (two leafs) `DecisionTreeClassifier`s, the importance of a feature will only be a count of how often it is selected as discriminating feature, _not_ of how well it helps discriminating (reducing impurity).

I would argue this is a problem in general (that is, also for other ensembles, for instance `RandomForestClassifier`), because it looses information on how important is the reduction in impurity provided by a given feature; but the problem is particularly important for boosting, because it implies attributing an equal importance to each weak learner, whereas given the sequential nature of these methods, being the most significant variable in the first or in the last weak learner doesn't really have the same meaning.

The [docs](https://scikit-learn.org/stable/modules/ensemble.html#forests-of-randomized-trees) refer to _G. Louppe, “Understanding Random Forests: From Theory to Practice”, PhD Thesis, U. of Liege, 2014_ for the definition of MDI, and in that work MDI is given in Equation (6.2), where I see no sign of normalization at the tree level (normalization happens at the ensemble level, simply by taking the mean across trees). On the other hand, the [API docs for AdaBoost](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html#sklearn.ensemble.AdaBoostClassifier.predict) refer to _Y. Freund, R. Schapire, “A Decision-Theoretic Generalization of on-Line Learning and an Application to Boosting”, 1995_ which doesn't seem to cover the problem of features importance.

So my understanding is that the current computation of feature importances in ensemble methods is not a proper implementation of the sources provided. This doesn't mean it doesn't necessarily make sense - more on this below.

#### Steps/Code to Reproduce


Example:
```python
import numpy as np
from sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier

N = 10**4
x = np.random.random((N, 10))

y = np.zeros(N, dtype=int)
# Make 0 the most important feature and 1 a less important one
y[:N//2] += x[:N//2, 0] > .5
y[3*N//4:N] += x[3*N//4:N, 1] > .5

tree = DecisionTreeClassifier(max_leaf_nodes=3)
tree.fit(x, y)
print("Importances in tree", tree.feature_importances_)
# ~.88, ~.012, followed by zeros

forest = RandomForestClassifier(max_leaf_nodes=3)
forest.fit(x, y)
print("Importances in forest", forest.feature_importances_)
# ~.58, ~.30, followed by numbers below 0.02

ada = AdaBoostClassifier(DecisionTreeClassifier(max_leaf_nodes=2), n_estimators=2, learning_rate=.5)
ada.fit(x, y)
print("Importances in ada", ada.feature_importances_)
# 0.5, 0.5, followed by zeros
```

#### Expected Results

What _should_ be done really is not obvious. I would say there are two main options:

1. stick to the definition provided by Louppe and mentioned above - basically, stop normalizing at the estimator level.
2. simply amend the current code in `AdaBoostClassifier` to weigh the nth estimator's by the nth power of `learning_rate`.

The difference between the two approaches is about whether we want importance to describe aspects of _data_ or of _prediction_: this is particularly clear for the Random Forest. Option 2. implies not changing anything: if in a given estimator all important variables are dropped, remaining variables will raise their importance because of normalization, but _this is precisely what will happen when predicting_, given that this estimator has weight 1 like all the others. In Option 1., instead, how prediction actually works is irrelevant: variables in a tree with a low discriminating ability will be simply assigned a low importance.

I'm happy to try to produce a patch, once it is decided (i) what is the desired behavior (ii) what kind of backwards compatibility we should guarantee.

#### Actual Results

```python
Importances in tree [0.87570382 0.12429618 0.         0.         0.         0.
 0.         0.         0.         0.        ]
Importances in forest [0.59394625 0.24287826 0.02311757 0.0115537  0.02783398 0.00115368
 0.04641229 0.00799708 0.03916944 0.00593776]
Importances in ada [0.5 0.5 0.  0.  0.  0.  0.  0.  0.  0. ]
```

#### Versions


System:
    python: 3.9.1+ (default, Jan 20 2021, 14:49:22)  [GCC 10.2.1 20210110]
executable: /usr/bin/python3
   machine: Linux-5.10.0-2-amd64-x86_64-with-glibc2.31

Python dependencies:
          pip: 20.1.1
   setuptools: 51.3.3
      sklearn: 0.24.1
        numpy: 1.19.5
        scipy: 1.6.0
       Cython: 0.29.21
       pandas: 1.3.0.dev0+381.g6259b5a737.dirty
   matplotlib: 3.3.3
       joblib: 0.17.0
threadpoolctl: 2.1.0

Built with OpenMP: True
user: toobaz
labels: Needs Decision: Requires decision
issue_19971: state: open
title: EHN Add transform_inverse to Nystroem
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
As discussed in ~#19732~ #19899, `sklearn.kernel_approximation.Nystroem` currently lacks `transform_inverse()` method unlike `PCA` and `KernelPCA`.

#### What does this implement/fix? Explain your changes.
This PR implements `transform_inverse` in `sklearn.kernel_approximation.Nystroem`.


#### Any other comments?
While implementing the method, I was not confident about the choice of linear system solver APIs. Maybe there is a better API for solving systems with a positive semidefinite matrix.

The formula I used for kernel ridge regression of original data points to the low-dimensional representation vectors are as follows (it is also commented in the code).
![image](https://user-images.githubusercontent.com/30857855/115966640-de349a00-a52e-11eb-951f-3ddebc4d993c.png)

The reconstruction performance was compared with KernelPCA using the example in #19945. The reconstruction quality seems identical in this case.
![Figure_3](https://user-images.githubusercontent.com/30857855/115966854-bd207900-a52f-11eb-84a3-36d86006e92b.png)
![Figure_2](https://user-images.githubusercontent.com/30857855/115966855-bdb90f80-a52f-11eb-81cb-251b507c04c8.png)




user: kstoneriv3
issue_19958: state: open
title: DOC Added references to the Matthews correlation coefficient function in the user guide
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.

I am proposing to include the reference to two articles of mine about the Matthews correlation coefficient because they describessome mathematical properties of MCC, some mathematical relationships between MCC and other measures ([F1 score and accuracy](https://doi.org/10.1186/s12864-019-6413-7); and [balanced accuracy, bookmaker informedness, and markedness](https://doi.org/10.1186/s13040-021-00244-z)), and some use cases about how it works and what message it generates.
I think this information might be useful for the scikit-learn users who would like to know more about MCC.
I already [proposed to include one of these two references in the MCC code function](https://github.com/scikit-learn/scikit-learn/pull/18707), but other users suggested me to add it to the user guide instead.

I am available to provide additional information, of course.

Thanks for considering my pull request.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: davidechicco
labels: Needs Decision: Requires decision
issue_19955: state: open
title: Warm starting as a "general trick"?
body: In https://scikit-learn.org/stable/developers/performance.html it is written

> The section A simple algorithmic trick: warm restarts gives an example of such a trick.

When I follow the link I come to https://scikit-learn.org/stable/developers/performance.html#warm-restarts which provides another link, which links to the Glossary in general. It **should** lead to https://scikit-learn.org/dev/glossary.html#term-warm_start - I guess. That is a first issue.

Secondly, while warm starting can be useful for some algorithms, I am not sure that it is obvious how it is most of the time useful. I admit though that often useful, but sometimes this is not so much a simple trick, but incorporating warm starting can be quite challenging (even though rewarding).

The issue I have with the documentation here, is that it gave the impression to me that it is a more or less obvious trick that should work most of the time.

But maybe there is to much interpretation on my part and that I am the only one getting this impression. If so, please ignore this. The reason I might be a bit biased is because I do academic foundational research on warm starting where it is not that easy.
user: Make42
issue_19941: state: open
title: Allow NMF to run on data that is not strictly nonnegative
body: Currently, if you call NMF on data with negative values, an error is thrown:

https://github.com/scikit-learn/scikit-learn/blob/95119c13a/sklearn/decomposition/_nmf.py#L1024

I do not believe there is a need for this restriction.

While it is more natural to apply NMF to nonnegative data, it is still well-defined if there are negative values. If the data has small negative values, one can threshold them to zero, but I think this is a choice the user should make on their own and not be forced into. Perhaps a warning could be thrown if negative values are encountered?

A (perhaps contrived) potential use case is if one wants to fit a model like...

```minimize  norm(X - W @ H - other_model.predict(X))```

where the matrices `W` and `H` are nonnegative factor matrices. In this case `X` is a nonnegative data matrix, but the residuals `X - other_model.predict(X)` are not nonnegative. One way of fitting the above model would be to alternate between fitting the NMF model and `other_model`:

```python
nmf = NMF()
other_model = OtherModel()

while not converged:
    W = nmf.fit_transform(X - Xhat)
    H = nmf.components_
    Xhat = other_model.fit_predict(X - W @ H)
```

Doing this would require fitting NMF to a residuals matrix with negative values.
user: ahwillia
issue_19938: state: open
title: TST add common test to check support for bytes target
body: Related to #18555
user: glemaitre
issue_19928: state: open
title: WIP KBinsDiscretizer supports NaN as input values
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
This is a fix to issue #19920 . Include the following changes:
- Add `force_all_finite=False`  in `self._validate_data`
- Propagate NaN values in `transform` and `inverse_transform` method

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: MaxwellLZH
issue_19926: state: open
title: SimpleImputer is very slow with DataFrames
body: `SimpleImputer` seems to scale badly with the number of rows when fitting it to a large `DataFrame` object, taking several times more than if using pandas' own functions in a loop.

Example:
```python
import numpy as np, pandas as pd
from sklearn.impute import SimpleImputer

rng = np.random.default_rng(seed=1)
nrows = int(1e6)
ncols = 10
nmissing = int(.1 * nrows)
X = pd.DataFrame(rng.normal(size=(nrows, ncols)))
for cl in X.columns:
    X.loc[rng.choice(nrows, nmissing, False), cl] = np.nan
```
```python
%%timeit
imputer = SimpleImputer(strategy="median").fit(X)
```
```
1.68 s ± 17.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```


Compare against this:
```python
from sklearn.base import BaseEstimator, TransformerMixin
class CustomImputer(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        self.medians = dict()
        for cl in X.columns:
            self.medians[cl] = X[cl].median()
        return self
    def transform(self, X):
        X_new = X.copy()
        for cl,m in self.medians.items():
            X_new[cl] = X_new[cl].fillna(m)
        return X_new
```
```python
%%timeit
imputer = CustomImputer().fit(X)
```
```
227 ms ± 2.61 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
user: david-cortes
issue_19920: state: open
title: KBinsDiscretizer doesn't handle NAs
body: If I try to pass some input to `KBinsDiscretizer` that would contain NaNs, it will throw an error instead of ignoring the NAs:
```python
import numpy as np
from sklearn.preprocessing import KBinsDiscretizer

X = np.arange(10).reshape((-1,1))
X[2] = np.nan
kb = KBinsDiscretizer(encode="ordinal", strategy="quantile")
kb.fit(X)
```
```
ValueError: cannot convert float NaN to integer
```

The bins could still be calculated if one ignores the NAs and outputs them as `np.nan` in the transformation.
user: david-cortes
issue_19914: state: open
title: WIP Adds Generalized Additive Models with Bagged Hist Gradient Boosting Trees
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

# This is very WIP!

**There are many things left to be done here! Please do not review yet!** Here is an [example](https://137869-843222-gh.circle-artifacts.com/0/doc/auto_examples/ensemble/plot_gam_boosted_regression_simple.html) of the estimator in action!

There is still much to be done for this PR, but I want to see if this fits in with our inclusion criterion. [Intelligible Models for Classification and Regression](https://www.cs.cornell.edu/~yinlou/papers/lou-kdd12.pdf) has 259 citations.

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Closes https://github.com/scikit-learn/scikit-learn/issues/3482

#### What does this implement/fix? Explain your changes.
This implementation builds on the parts of `HistGradientBoosting*` by restricting the splitter to only split one feature at a time.

[Intelligible Models for Classification and Regression](https://www.cs.cornell.edu/~yinlou/papers/lou-kdd12.pdf) shows that Bagged Gradient Boosting Trees are better than the other GAMs (backfilling techniques or using splines). The tree based approach was shown to train faster for bigger datasets.

Using the histograms is based off of a follow up paper [Accurate Intelligible Models with Pairwise Interactions](https://www.cs.cornell.edu/~yinlou/papers/lou-kdd13.pdf). In this paper, they also describe a way to quickly obtain pairwise interactions. (This can be a follow up PR)

CC @amueller 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
issue_19907: state: open
title: Dummyclassifier with binary and multiclass and with predict proba
body: #### Describe the workflow you want to enable
There is case of mixing multiple classifiers with SelectFirstClassifier. This allows to include dummy constant classifiers for specific features conditions. For example this allows to provide fixed (rules based) prediction for some combination of features and their thresholds. Unfortunately to export pipeline with such scenario requires all classifiers to have the same output shape and structure. There are workarounds, but much better us to have dummy classifier with constant multiclass output with probabilities set with parameters. 

#### Describe your proposed solution
Add support of binary and multiclass predictions with parameters controlling predict proba output.

#### Describe alternatives you've considered, if relevant
There is a way to add dummy records into training data and use real multiclass classifiers in combination with feature filters/droppers and selectfirstclassifier. This approach is not very stable and does not provide that much controll as defining exactly output probabilities.

There might be other ways to do controlled multyclass output for specific feature/thresholds combination in addition to main classifiers of pipeline and export it as pmml model. Please let me know about it. Thank you.

#### Additional context
Maybe ability to controll output probabilities with input features using function transformers with callables might be even better and broader use case solution.

user: petr-ponomarenko
issue_19899: state: open
title: Implement Nystroem.inverse_transform
body: Our `KernelPCA(fit_inverse_transform=True)` implements the method of:

Learning to Find Pre-Images
Neural Information Processing Systems 16 (NIPS 2003)
Jason Weston, Bernhard Schölkopf, Gökhan Bakir

https://papers.nips.cc/paper/2003/hash/ac1ad983e08ad3304a97e147f522747e-Abstract.html

We might want to adapt this to make `make_pipeline(Nystroem(), PCA())` a feature complete, scalable alternative to `KernelPCA` for denoising applications and novelty detection for instance.
user: ogrisel
issue_19898: state: open
title: Improve near constant feature detection in scalers
body: To avoid scaling near constant features by a very large value, a heuristic was introduced in #19527. However it was a bit too absolute (e.g. #19726). It was fixed for StandardScaler, but there might be similar issues for the other scalers.

It should be tested case by case, depending on how the scale is defined. To see if the heuristic is appropriate or not, it should be tested against a very wide range of scaling of the data. If it breaks similarly to #19726, we need to find an appropriate bound on the scale to declare the feature constant (as it was done in #19788 for StandardScaler)
user: jeremiedbb
issue_19896: state: open
title: PLSRegression include percentage of variance in Y explained by each component
body: #### Describe the workflow you want to enable
I would like the PLSRegression algorithm to give me the percent variance explained in Y by each component. Similar to the PCTVAR in MATLAB's plsregress function, see: https://www.mathworks.com/help/stats/plsregress.html

#### Describe your proposed solution
Depending of whether scale=True or scale=False, I am able to calculate the explained variance in Y using the following data file:
[example_data.xlsx](https://github.com/scikit-learn/scikit-learn/files/6311002/example_data.xlsx)
and the Python code below.

What I am requesting is for the PLSRegression to provide this information directly, without having me to calculate it
`
#%% Packages

import pandas as pd
import numpy as np
from sklearn.cross_decomposition import PLSRegression

import numpy as np

#%% Import Data

data_filepath = 'example_data.xlsx'

data = pd.read_excel(data_filepath, sheet_name='data')

#%% PLS Parameters

pls_n_components = 4 # use 4 components

pls_scale = False # no scaling


#%% PLS Regression

X = data[["N1","N2","N3","N4"]] # predictors
Y = data[["Y"]] # response


pls = PLSRegression(n_components=pls_n_components, scale=pls_scale)
pls.fit(X,Y)

if pls_scale:
    pls.intercept_ = pls._y_mean
else:
    pls.intercept_ = pls._y_mean - np.dot(pls._x_mean , pls.coef_)

Y_pred = pls.predict(X)

from sklearn.metrics import r2_score
from sklearn.metrics import mean_squared_error

r2_sum = 0
for i in range(0,pls_n_components):
    if pls_scale: # if scaling done
        Y_pred2=np.dot(pls.x_scores_[:,i].reshape(-1,1),pls.y_loadings_[:,i].reshape(-1,1).T)*Y.std(axis=0, ddof=1)[0]+Y.mean(axis=0)[0]
    else:
        Y_pred2=np.dot(pls.x_scores_[:,i].reshape(-1,1),pls.y_loadings_[:,i].reshape(-1,1).T)+Y.mean(axis=0)[0]
        
    r2_sum += round(r2_score(Y,Y_pred2),3) 
    print('R2 for %d component: %g' %(i+1,round(r2_score(Y,Y_pred2),3)))
   
print('R2 for all components (): %g' %r2_sum) #Sum of above
print('R2 for all components (): %g' %round(r2_score(Y,pls.predict(X)),3)) #Calcuted from PLSRegression's 'predict' function.
`


#### Describe alternatives you've considered, if relevant

#### Additional context

user: helsharif
issue_19880: state: open
title: RFC shape of output of predict_proba for multilabel-indicator
body: In https://github.com/scikit-learn/scikit-learn/pull/19859, I am heading toward addressing some issue already reported long time ago there: https://github.com/scikit-learn/scikit-learn/issues/2451

We have a range of classifier supporting multilabel classification. To give a bit of feedback, I recall what a multilabel indicator matrix look like:

```python
In [1]: from sklearn.datasets import make_multilabel_classification

In [2]: X, y = make_multilabel_classification(n_samples=10)

In [3]: y
Out[3]: 
array([[0, 0, 0, 0, 0],
       [0, 0, 0, 0, 1],
       [0, 1, 0, 0, 1],
       [0, 1, 0, 0, 1],
       [1, 1, 0, 0, 0],
       [0, 1, 1, 0, 0],
       [0, 1, 0, 0, 0],
       [0, 0, 1, 0, 0],
       [0, 1, 0, 0, 0],
       [0, 1, 1, 1, 1]])
```

Each column contain 0/1 entries and you can have several 1 for each row. This type of classification is a special case of `multiclass-multioutput` where each column will not be limited to 0/1 entries but to multiclass labels.

We have some issue with consistency in scikit-learn when calling the method `predict_proba` with classifiers handling these type of data.

## Current situation in scikit-learn

We have a couple of classifiers that handle multilabel targets: tree-based models, neighbors, MLP, and ridge. In addition, the tree-based models and neighbors classifiers handle `multiclass-multioutput`.

In a `multiclass-multioutput` problem, `classifier.predict_proba` return a list of n_outputs containing arrays of shape (n_samples, n_classes). Since multilabel classification is a specific case of `multiclass-multioutput`, `classifier.predict_proba` will return a list of n_outputs containing arrays of shape (n_samples, 2). This should be an historical choice.

MLP and ridge (currently broken - see https://github.com/scikit-learn/scikit-learn/pull/19869) do not support `multiclass-multioutput`. There internal design make that it is easy to output an array of shape (n_samples, n_outputs) containing the probability of the positive class only when calling `classifier.predict_proba`.

Therefore we have two prediction format for the classifier. This RFC intends at taking a decision regarding this discrepancy.

## Potential solutions

### 1. Keep the outputs as is

We can decide to keep the output as we currently do. One can argue that by supporting `multiclass-multioutput` it is natural to keep the extensive format list of arrays while being efficient with arrays only for classifiers supporting `multilabel-indicator` targets.
It would require to make the documentation to be explicit.

### 2. Make output consistent

#### 2.1 By adding an estimator parameter

We could add an estimator parameter allowing to switch from one representation to another.

### 2.2 By adding a parameter to the `predict_proba` method

We could add a parameter to the `predict_proba` method to switch the output of the representation. It might still imply a classifier to store which type of target where fitted to distinguish between `multiclass-multioutput` and `multilabel-indicator` at predict.

### 2.3 By forcing all estimators to a specific format

In addition, we could enforce a specific format such that by default all classifiers return the same output format. In this case, it requires some deprecation cycles.
user: glemaitre
issue_19877: state: open
title: Common test are not run for the encoders
body: Related to #14057

The encoders inheriting from `BaseEncoder` get the tag `X_type=["categorical"]`. Since that `X_type` does not contain `"2darray"`, we are bypassing a lot of tests.

Notably we don't check for `n_features_in_` after `fit` currently.
user: glemaitre
labels: module:test-suite: everything related to our tests
issue_19874: state: open
title: ENH add parameter in accuracy_score for multilabel classification
body: closes #16832

Propose to add a parameter `multilabel` to `accuracy_score` to compute either the exact match ratio or the Hamming score.
user: glemaitre
issue_19868: state: open
title: ElasticNet (and Lasso) for multiple outputs is not described correctly
body: #### Describe the issue linked to the documentation

There are 2 ways to run ElasticNet with multiple outputs. The first is to run it for each output (method A) and the second is to use a different objective, using `||W||_21` as norm (method B). From what I see, ElasticNet is supposed to apply method A and MultiTaskElasticNet uses method B. enet_path and lasso_path apply method B.

However, ElasticNet and Lasso's path method is exactly enet_path (`path = staticmethod(enet_path)`). This causes confusion, as

1. fit and path don't apply the same algorithm
2. method B is documented in by enet_path and hence auto-included on the pages of ElasticNet and Lasso

#### Suggest a potential alternative/fix

Changing the API for `path` to match method A would cause compatibility issues. I propose to add a note somewhere to explain that `path` doesn't apply the same algorithm as `fit` in Lasso and ElasticNet, and add a link to MultiTaskLasso and MultiTaskElasticNet.

user: louisabraham
issue_19854: state: open
title: [WIP] Adding Local Regression (Fixes #3075)
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #3075 

#### What does this implement/fix? Explain your changes.
The above issue is that local regression is missing in scikit-learn. This PR addresses this issue by adding an implementation of local regression and expanding the test suite to cover this.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: niknow
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_19853: state: open
title: ClassifierChain should only accept multilabel-indicator
body: By trying to solve https://github.com/scikit-learn/scikit-learn/issues/19357 and writing some test, it seems that `ClassifierChain` is expected to be fitted with `multilabel-indicator` target (each column should only contain 0/1 classes).

However, there is no check and one can fit a `multiclass-multioutput` target. The classifier will later fail if calling `decision_function` that would return an array of shape `n_samples, 3` while an array of `n_samples,` is expected. I assume a similar behaviour for `predict_proba`.

I think that we should check the type of target to raise a proper error at `fit`.
user: glemaitre
issue_19851: state: open
title: quantile regression with dynamic or multiple quantiles
body: Is anyone thinking of extending GBR to include multiple quantiles (and the monotonicity constraint)?

It is nice to see Quantile Regression is making it's way in sklearn for example here

https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor

We can also do quantile regression as a function approximator f(tau, X) where X is the feature context and tau is a specific probability requested (and input into the particular loss function). 

The factor that complicates this is that one really needs to bake monotonicity into the approximator $f$. This can be done in a few simple ways or more generally but is probably approximator dependent.

A simplest first discussion point might be in extending the GBR to take a fixed grid of quantiles as I suspect this would simply add a loop to the loss and the regressor could be constrained to be positive outputs that are then cumulative summed to get a monotic structure in $\tau$. 

I have not described this cleary but hope someone might get the gist of this and can point me in the right direction or discuss further.

Noting this maybe should go with https://github.com/scikit-learn/scikit-learn/issues/18997
user: cottrell
issue_19848: state: open
title: CalibratedClassifierCV based on a prefitted model used as input to SelfTrainingClassifier results in an exception
body: #### Describe the bug

Using a CalibratedClassifierCV model based on a prefitted model (e.g. a RandomForestClassifier as in the example below) as input to a SelfTrainingClassifier model yields the following error:
```
sklearn.exceptions.NotFittedError: This RandomForestClassifier instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.
```

#### Code to Reproduce

```python
import numpy as np
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.semi_supervised import SelfTrainingClassifier

X, y = make_classification(n_samples=1000, n_features=4, n_redundant=0, random_state=42)
X2, X_cb, y2, y_cb = train_test_split(X, y, test_size=0.2, random_state=42)
X_unlabelled, X_labelled, y_unlabelled, y_labelled = train_test_split(X2, y2, test_size=0.25, random_state=42)
y_unlabelled[:] = -1

clf = RandomForestClassifier(max_depth=4, random_state=42)
clf.fit(X_labelled, y_labelled)

calibrated_model = CalibratedClassifierCV(base_estimator=clf, cv="prefit")
calibrated_model.fit(X_cb, y_cb)

X_final = np.concatenate((X_unlabelled, X_labelled), axis=0)
y_final = np.concatenate((y_unlabelled, y_labelled), axis=0)

self_training_model = SelfTrainingClassifier(calibrated_model)
self_training_model.fit(X_final, y_final)
```

#### Versions
python: 3.8.8
machine: Windows-10

Python dependencies:
pip: 21.0.1
setuptools: 52.0.0.post20210125
sklearn: 0.24.1
numpy: 1.19.2
scipy: 1.6.2
Cython: 0.29.22
pandas: 1.2.3
matplotlib: 3.3.4
joblib: 1.0.1
threadpoolctl: 2.1.0

user: Z3pherus
labels: Needs Decision - API: Requires decision regarding API
issue_19838: state: open
title: Request TNR & NPV to be added to the Classification Report
body: The `sklearn.metrics.classification_report` outputs 
- `precision (PPV)`
- `recall (TPR)`
- `f1-score`
- `support`

For `multi-class` dataset, the `sklearn.metrics.classification_report` will also calculate their `micro/macro` averaging, which is really nice. 2 other metrics 
- `NPV (negative predictive value)`
- `TNR (specificity)`

are also very important. Every time I evaluate new classifier I have to manually calculate `NPV` and `TNR`, so it would be awesome if `sklearn.metrics.classification_report` also outputs them as well.

Would also be nice to have the following as well
- `AUROC`
- `AUPR`
- `MCC`

user: Scoodood
issue_19832: state: open
title: Should we use numpy qr rather than scipy's for faster decomposition when matrices are big ?
body: I came across this topic while working on randomized methods for decomposition (#12069).

I found [this stackoverflow post](https://stackoverflow.com/questions/34281737/numpys-qr-appears-faster-than-scipys-why) challenging the speed of Numpy vs. Scipy QR decompositions (based on the same fortran engine but probably using different options or strategies) 

It made me create an experiment (see [this answer](https://stackoverflow.com/a/66970690/7262247)) to find out that maybe (?) there is actually some difference for large matrices.
I am not sure if this should be a concern for scikit-learn, but at least it is now properly logged with an associated experiment :) .

![image](https://user-images.githubusercontent.com/3236794/113727364-5a3c7e80-96f5-11eb-9b5c-fbf669db1d47.png)


user: smarie
issue_19830: state: open
title: LabelEncoder ValueError on unseen labels is too broad.
body: Hi,

When `LabelEncoder` encounters unseen labels it raises `ValueError` :arrow_down: 

https://github.com/scikit-learn/scikit-learn/blob/114616d9f6ce9eba7c1aacd3d4a254f868010e25/sklearn/utils/_encode.py#L180

I'm wondering if it couldn't be nice to have a more specific exception instead, like a custom `UnseenLabelException`. 
There are some cases where one could want to catch this specific exception in order to trigger something (like a retraining job with new fresh labels).

Right now, to catch this behaviour the only thing I came up with is `except ValueError as err:` and then checking some regex matches with `err.args`. Which feels a bit shaky somehow.
user: dithyrambe
issue_19802: state: open
title: Add a "Copy Import" button to the top of each importable module's documentation page
body: #### Describe the workflow you want to enable

Attempting to use different Sklearn models and features requires rapidly switching between pages on the documentation and importing many different objects/modules. As far as I know, the two easiest ways to copy the import statement from a module's documentation page (e.g. `from sklearn.linear_model import LinearRegression`) is to double click the title (e.g. sklearn.linear_model.LinearRegression) or select the import statement in the "Examples" section, then copy and paste it into your IDE. Both of these are not bad, but add maybe 10-20 seconds to each import. If there were a button on the top of the page (near or next to the title, perhaps) which when clicked copied the import line, then the time saved would really add up.

#### Describe your proposed solution
A button near the top of the page, maybe next to the title which copies the import statement string so you can paste it into your IDE.

Example: ![copy_suggestion](https://user-images.githubusercontent.com/11083975/113237969-dc502180-9275-11eb-9fe2-3fa0e3d9027d.png)

Sorry for the less than ideal design.

#### Describe alternatives you've considered, if relevant

The button could go somewhere else.

Instead of a button to copy the text, the import statement could be listed at the top of the page

#### Additional context

I often find the modules I am looking for by googling them instead of by just typing them in directly to my IDE.

Thanks!
user: MaxNiebergall
issue_19789: state: open
title: DOC Add warnings about causal interpretation of coefficients
body: #### Reference Issues/PRs
Partially addresses #19413 

#### What does this implement/fix? Explain your changes.
Add some commentary on why it is difficult to infer causality from a predictive model due to confounding from unobserved variables. Also add a brief comment on "no causation without manipulation".

#### Any other comments?
I also wanted to make some additional comments in what situations it would be appropriate to make a causal deduction but was afraid that it would be too verbose. 
user: hxyue1
issue_19754: state: open
title: Draft implementation of non-parametric quantile methods (RF, Extra Trees and Nearest Neighbors)
body: Dear scikit-learn maintainers and contributers,

this PR aims to provide a draft implementation for Random Forest Quantile Regression (https://jmlr.org/papers/volume7/meinshausen06a/meinshausen06a.pdf) and KNN Quantile Regression (https://www.tandfonline.com/doi/abs/10.1080/02664763.2015.1070807?journalCode=cjas20&journalCode=cjas20). Both methods are widely used, and would be an interesting addition to scikit-learn as they can estimate conditional quantiles without having to fit each quantile separately (thus providing significant speedups). I started the development of these algorithms from the implementation of scikit-garden, which seems to not be maintained anymore.

#### Reference Issues/PRs

Fixes #11086
Takes some steps of #18997 

#### What does this implement/fix? Explain your changes.

- Random Forest Quantile Regression
- Extra Trees Quantile Regression (which uses the same implementation as QRF)
- KNN quantile regression
- A weighted_quantile function, which aims to mimic np.quantile with the inclusion of a weight parameter.

#### Any other comments?

This is very much a draft PR. Although the code is documented, I did not yet create the actual documentation for these additional algorithms (as the API is not yet fixed). Additionally, if these methods are approved the partial_dependence functions need to be changed (to pass the quantile parameter to the predict function). 

Note that some parts are currently optimised with numba, rather than Cython. I am still attempting to do this conversion myself, but any help on this would be great.

user: jasperroebroek
issue_19753: state: open
title: FIX VotingClassifier handles class weights
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
This is a fix to issue #18550 

#### What does this implement/fix? Explain your changes.
We look into the base estimators and encode the `class_weights` if they are provided.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: MaxwellLZH
issue_19751: state: open
title: [MRG] Add min_tpr parameter to roc_auc_score, allow both min_tpr and max_fpr 
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes #11668
Fixes #11725
See also #3273
See also #3840

#### What does this implement/fix? Explain your changes.
- Add min_tpr option for sklearn.metrics.roc_auc_score. Make the function allowing both `min_tpr` and `max_fpr` limitations at the same time. 
- Add unit test and common tests for the new argument
- Add an example documentation to explain the standardized partial AUC calculation

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Fengjun-Wang
issue_19746: state: open
title: FIX Ridge.coef_ return a single dimension array when target type is not continuous-multiple
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
This is a fix to issue #19693 following the discussion from @thomasjpfan 

#### What does this implement/fix? Explain your changes.
Both `Ridge.coef_` and `RidgeCV.coef_` now returns a single dimension array when the result of `sklearn.utils.multiclass.type_of_target(y)` is continuous.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: MaxwellLZH
issue_19744: state: open
title: Make Isomap to work with any precomputed neighborhood graph, at least radius_neighbors_graph
body: #### Describe the workflow you want to enable
Now Isomap assumes only k-nearest neighbors neighborhoods, even if supplied with precomputed neighborhood graph, such as radius_neighbors_graph which by definition have an arbitrary number of neighbors for each point (those which are within provided eps-radius).

When supplied with precomputed matrix of eps-radius nearest neighbors Isomap gives error:

```python
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler
from sklearn.manifold import MDS, Isomap

from sklearn.neighbors import radius_neighbors_graph

X, y = load_digits(return_X_y=True)
X = StandardScaler().fit_transform(X)

X, y = load_digits(return_X_y=True)
A_sparse = radius_neighbors_graph(X, radius=5.5, mode="distance")
isomap = Isomap(metric="precomputed")
isomap.fit(A_sparse)
```

> ValueError: 6 neighbors per samples are required, but some samples have only 0 neighbors in precomputed graph matrix. Decrease number of neighbors used or recompute the graph with more neighbors.

#### Describe your proposed solution
Make Isomap work with supplied precomputed neighborhood graphs, with an arbitrary number of nearest neighbors.


user: oleg-kachan
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_19731: state: open
title: ENH Improve regularization messages for QuadraticDiscriminantAnalysis
body: #### Reference Issues/PRs
References #14997

#### What does this implement/fix? Explain your changes.
Remove warning: `the variables collinear`.
Check the covariance matrix after regularization and raise a LinAlgError prompting to increase regularization. 
I left the `tol` argument as a way to control the error. The user can turn it off if need be.
Updated the documentation for `tol` argument.
Change the tests to check for the correct errors.

#### Any other comments?
The last test was to check for cases where n_samples_class < n_features, I tried many different configurations but I was unable to produce a set of variables the produced an output that didn't throw an error. The test is only there with negative examples, I am open to suggestions for any positive examples in this case.

user: azihna
issue_19719: state: open
title: skip check_classifiers_predictions test for poor_scoor estimators
body: #### Reference Issues/PRs
This will fix #16799


#### What does this implement/fix? Explain your changes.
This will skip a part of the check_classifiers_predictions test if it's a poor_score estimator.
Currently, the test is only skipped if it a ComplementNB Classifier. 

user: not-a-feature
issue_19717: state: open
title: Transform method for TSNE (different from the fit_transform already present)
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
The t-SNE embedding class has a fit_transform method, and a fit method, but doesnt contain a transform method. This in my understanding, is due to the fact that t-SNE is supposed to be used to create embeddings for unsupervised data, and unsupervised data may not have a clear train-test division.  
However, consider the case where one would want to use the t-SNE embedding as an added feature to feed into some machine learning model / pipeline. In this case, it would be beneficial to have a transform method to obtain the embeddings of the test data points, alternative being run t-SNE on the entire (test + train), which in near to ideal cases may work, but might take up too much inference time, or be subject to unduly changes in the data (test + train) distribution.

#### Describe your proposed solution
A solution, somewhat inspired by the Euclidean norms in the t-SNE algorithm math, would be as follows:
* Required : the train_data, test_datapoint (just one point, can be scaled up later), train_embeddings
* Algorithm:
    * Calculate the top k Euclidean distances between the test_datapoint and all the points in the train_data
    * Get the embeddings of these previous top k data points train_data 
    * test_embedding can then be an average of these top k train_embedding points, weighted by the top k distances calculated in the first step
    * Repeat for all the data points 
 

#### Describe alternatives you've considered, if relevant
No other alternatives considered yet, but maybe some ingenuity can be put into how exactly to weight the top k embedding points in step 3 in the above described algorithm.   
Some solutions could be - 
* Weight them as 1/distance, inspired from the KNN algorithm
* Weight them as e^(-distance). A problem encountered here was that e^(-distance) might need a lot of precision if distance is really large, and hence, can be a memory intensive task.

#### Additional context
Recently encountered this in a fundamental data science problem me and my colleagues were working on. This might be really helpful to us, and probably can open up avenues to other data scientists as well.
user: hades-rp2010
issue_19710: state: open
title: Introduce honest sampling into sklearn RandomForestClassifier
body: **Is your feature request related to a problem? Please describe.**

Honest sampling in forests was first outlined by Breiman et al. 1984 in Classification and Regression Forests, in which it was suggested that random forests improve performance when the feature space is partitioned on one subset of samples, and the posteriors are estimated on a disjoint subset of the samples. This idea was revived by Denil et al. as structure vs. estimation points, and clarified and implemented by Wager and Athey. They identified several benefits of honest sampling: reduced bias, centered confidence intervals, reduced mean squared error, and the possibility of building causal forests.

From Wager and Athey 2018 (section 2.4: Honest Trees and Forests):

> In our discussion so far, we have emphasized the flexible nature of our results: for a wide variety of causal forests that can be tailored to the application area, we achieve both consistency and centered asymptotic normality, provided the subsample size s scales at an appropriate rate. Our results do, however, require the individual trees to satisfy a fairly strong condition, which we call honesty: a tree is honest if, for each training example i, it only uses the response Yi to estimate the within-leaf treatment effect τ using (5) or to decide where to place the splits, but not both. 

In addition, Wager and Athey note that subsampling does not "waste" training data:

> However, in our case, the forest subsampling mechanism enables us to achieve honesty without wasting any data in this sense, because we rerandomize the I/J-data splits over each subsample. Thus, although no data point can be used for split selection and leaf estimation in a single tree, each data point will participate in both I and J samples of some trees, and so will be used for both specifying the structure and treatment effect estimates of the forest. Although our original motivation for considering double-sample trees was to eliminate bias and thus enable centered confidence intervals, we find that in practice, double-sample trees can improve upon standard random forests in terms of mean-squared error as well.

**Describe the solution you’d like**


[EconML](https://github.com/microsoft/EconML/) has forked scikit-learn to create honest trees and generalized random forests for causal questions. We intend, instead, to merge back into scikit-learn based on the insights from EconML in regression trees, while also building classification trees, with the ability to accept both dense and sparse data.



_Key references:_

Breiman L, Friedman J, Stone C, Olshen R. Classification and Regression Forests. 1984.

Denil M, Matheson D, De Freitas N. Narrowing the Gap: Random Forests In Theory and In Practice. Proc 31st Int Conf Mach Learn. 2014; 665–673. Available: http://jmlr.org/proceedings/papers/v32/denil14.html

Wager S, Athey S. [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://arxiv.org/abs/1510.04342). J Am Stat Assoc. 2018;113: 1228–1242. doi:10.1080/01621459.2017.1319839

Guo R, Mehta R, Arroyo J, Helm H, Shen C, Vogelstein JT. Estimating Information-Theoretic Quantities with Uncertainty Forests. 2019; 1–19. Available: http://arxiv.org/abs/1907.00325

user: EYezerets
issue_19707: state: open
title: ENH BestSplitter split on feature with lowest index when multiple features have identical order
body: #### Reference Discussion
Tree models (DecisionTree, RandomForest, GradientBoosting) "splits" feature importance between features with identical order, which does not happen in XGBoost or LightGBM.

More details and code example can be found in [this](https://github.com/scikit-learn/scikit-learn/discussions/19569) discussion thread.

#### Explanation
When different splits have the same improvement, we pick the one with the lowest feature index.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: MaxwellLZH
issue_19698: state: open
title: Adding min-max linkage to Agglomerative Clustering
body: Hi,

I checked out the "the scikit-learn inclusion criterion:" and the algorithm I am about to suggest satisfies all except one! And, that's the number of  citations, which is 116 according to google scholar (not  200+). 

However, I think many people (who are on the application side of ML) use scikit-learn for their projects and are not aware of many algorithms that are available. Thus, including a new algorithm in the scikit-learn can let them know such method exists. Please let me know if it is a good idea to add this method can be added to the Agglemorative Clustering. If so, I can work on it (as I already implemented it myself) and add it to the package. 


Challenge: The linkages in agglemorative clustering that can work with the distance matrix are complete, linkage, and average. However, the resulted clusters don't have a clear centroid. So, a method that can work with a distance matrix and provide a centroid-based structured clusters can provide a lot of flexibility. It can  be used for any distance measure and since it utilizes the distance matrix only, it can be performed very fast for large-size data. It also provides centroids which can be used as the representatives of the clusters.

Solution: A min-max linkage proposed by Jacob Bien & Robert Tibshirani (2011) is a good solution. It merges two clusters if a ball that is required to cover the points of the two clusters together is the smallest compare to others. A ball for each cluster can be identified as follows: First, identify the furthest neighbor of each point and calculate its distance. Then, choose the point (M) that has the shortest distance to its furthest neighbor. That point M is the centroid and that shortest distance is the radius of the ball. I recently used it for my paper where I had to use DTW distance, but I also need a fast algorithm that provides a reasonable centroid rather than computing DBA at each step of the process.

Alternative solution: N/A

Additional Info: 
**Paper:** 
Bien, Jacob, and Robert Tibshirani. "Hierarchical clustering with prototypes via minimax linkage." Journal of the American Statistical Association 106.495 (2011): 1075-1084.

user: NimaSarajpoor
issue_19693: state: open
title: Ridge and Lasso return different shaped .coef_ attributes
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
<!--
-->
`Ridge.coef_` returns an array with shape `(1, n_features)`, while `Lasso.coef_` returns an array with shape `(n_features,)`. LinearRegression.coef_ resembles Ridge.coef_ 

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
docs = ["Help I have a bug" for i in range(1000)]
vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)
lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->
```python
import numpy as np
from sklearn.linear_model import Ridge, Lasso
x = np.array([1,2,3,4,5]).reshape(-1,1)
y = np.array([2,4,6,8,10]).reshape(-1,1)
ridge_reg = Ridge()
lasso_reg = Lasso()
_ = ridge_reg.fit(x,y)
_ = lasso_reg.fit(x,y)
print(ridge_reg.coef_.shape)
print(lasso_reg.coef_.shape)
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
```python
(1, 1)
(1, 1)
```
#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
```python
(1, 1)
(1,)
```
#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
import imblearn; print("Imbalanced-Learn", imblearn.__version__)
-->
System:
    python: 3.7.6 (default, Feb 14 2020, 16:17:39)  [Clang 11.0.0 (clang-1100.0.33.17)]
executable: ./venv-mmm/bin/python
   machine: Darwin-19.6.0-x86_64-i386-64bit

Python dependencies:
          pip: 21.0.1
   setuptools: 41.2.0
      sklearn: 0.23.2
        numpy: 1.19.1
        scipy: 1.5.2
       Cython: None
       pandas: 1.1.1
   matplotlib: 3.3.1
       joblib: 0.16.0
threadpoolctl: 2.1.0

Built with OpenMP: True

<!-- Thanks for contributing! -->

user: mjcronin
issue_19679: state: open
title: Pass the confusion matrix as a parameter
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable

Functions like sklearn.metrics.balanced_accuracy_score, f1_score, jaccard_score, matthews_corrcoef, etc. recalculate the confusion matrix every time, which is very time consuming. I would want to calculate it once and then pass it to these functions as a parameter to get the resulting quality metrics much faster. If you want to calculate more than one of these scores on the same big data set this will save a lot of time.

#### Describe your proposed solution

Check if the parameter "cm" is passed and then skip its recalculation if yes.

user: saskra
issue_19665: state: open
title: Inconsistent behavior with bagging & base estimator class_weight
body: #### Describe the bug

Bagging Classifier transforms y to indices between 0..N but does not impact the possible `base_estimator.class_weight` attribute making the fit method fails when using class_weight: the class could not be found.

#### Steps/Code to Reproduce

This is working fine: 

```python
from sklearn.datasets import make_classification
from sklearn.linear_model import PassiveAggressiveClassifier

X, y = make_classification(n_samples=10000, n_features=5, n_redundant=0, n_clusters_per_class=1, weights=[0.5])
# let's say y is not 0 or 1 but something else like 1 and 2 for instance:
y += 1
# now i fit a classifier with class_weight, works fine
PassiveAggressiveClassifier(class_weight={1: 1, 2: 1}).fit(X, y)
```

Now i'm just wrapping it around BaggingClassifier:

```python
from sklearn.ensemble import BaggingClassifier

BaggingClassifier(PassiveAggressiveClassifier(class_weight={1: 1, 2: 1})).fit(X, y)
```

I'm getting: 

```pytb
ValueError: Class label 2 not present.
```

It seems it comes from here : https://github.com/scikit-learn/scikit-learn/blob/95119c13af77c76e150b753485c662b7c52a41a2/sklearn/ensemble/_bagging.py#L654 

Because bagging transform y but then class_weight attribute of the base estimator is not aligned with the new labels.

Is it expected or this is a bug ?

#### Versions

```
 System:
    python: 3.7.9 (default, Feb 13 2021, 00:48:23)  [GCC 10.2.1 20210110]
executable: /home/.../.pyenv/versions/3.7.9/bin/python3.7
   machine: Linux-5.10.0-3-amd64-x86_64-with-debian-bullseye-sid

Python dependencies:
          pip: 21.0.1
   setuptools: 53.0.0
      sklearn: 0.24.1
        numpy: 1.20.1
        scipy: 1.6.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```
user: TTRh
issue_19656: state: open
title: Variable regularisation strengths for Lasso Ridge and ElasticNet
body: #### Describe the workflow you want to enable
I would like to be able to set different alpha (regularisation strengths) to different features (columns of x). I've done this often for models I've made in CVXpy but it would be nice to have it in sklearn with all of the computation advantages sklearn has

#### Describe your proposed solution
This could be done by having the regularisation strength be a vector of length `p` (if `p = X.shape[1]`)

#### Describe alternatives you've considered, if relevant
An alternative to this, which is not computation very effective, is to have a model with a subset of the features of `X` and different regularization strengths all train on each other residuals in a loop until converges.

#### Additional context
N/A

I hope this was clear enough. Thanks folks. 
user: arose13
issue_19649: state: open
title: Optimizing the fine-tuning of PCA / KernelPCA n_components
body: This example shows a way to use cross-validation to select the best value for n_components in the PCA:
https://scikit-learn.org/stable/tutorial/statistical_inference/putting_together.html

However, it seems that there would be a more computation-efficient way to do it, fitting the PCA only once with the maximum value of n_components tested, and then applying the transformation with different values of n_components.
Indeed, when using GridSearchCV to optimize n_components in the PCA, there is a new fit for each iteration.
As the PCA can be very slow, optimizing it would be cool. 

Any idea on how to do it easily with current implementation of PCA and GridSearchCV ?


user: dcoulomb
issue_19644: state: open
title: Implemented persistent random seeds for OOB score calculation in random forests
body: This is an attempt to fix issue #19632 where OOB score calculations appear to randomly generated instead of the same ones that were used when training the tree. 

#### Changes
The random forest ``fit(..)`` function now generated a random ``seeds_for_bootstrapping_`` list that is used to initialise the a random generator that draws random indices for each tree.

#### Issues
The current code results in testing failures. I am nevertheless opening a pull request on the suggestion of @NicolasHug, to assist with debugging.
user: simontindemans
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_19632: state: open
title: OOB calculation in Random Forests seems to use incorrect indices
body: ### update 9 March 2021

After the discussion below and further testing, I came to the conclusion that this is _**not**_ a bug after all. The indices are computed correctly, because ``_make_estimator`` always generates a new random _seed_ that is stored in the ``random_state`` variable of each tree -- even if a ``np.random.RandomState()`` object was passed to ``_make_estimator``.

Apologies for the false alarm. I got confused by the interchangeable use of ``random_state`` as a seed or a random generator (but, interestingly, not as the actual state of the random generator). 

### original report

I have been looking into the OOB score calculation for random forests, and I am concerned that the indices considered to be "out-of-bag" are not actually the same as those held out for training. A few years ago, PR https://github.com/scikit-learn/scikit-learn/issues/4783 was implemented, which replaced the explicit storage of indices by a regeneration procedure.

Specifically, this resulted in the following code fragment in ``_compute_oob_predictions(self, X, y)`` in ``sklearn/ensemble/_forest.py``:

    for estimator in self.estimators_:
        unsampled_indices = _generate_unsampled_indices(
            estimator.random_state, n_samples, n_samples_bootstrap,
        )

        y_pred = self._get_oob_predictions(
            estimator, X[unsampled_indices, :]
        )
        oob_pred[unsampled_indices, ...] += y_pred
        n_oob_pred[unsampled_indices, :] += 1

If ``estimator.random_state`` is a np.RandomState object instead of a constant seed then this appears to generate new random indices instead of the original set of indices. I noticed that this potential shortcoming was also observed by @amueller in the discussion about https://github.com/scikit-learn/scikit-learn/issues/4783, but no follow-up was included there:

> Silly question: This fails if a `np.RandomState` was passed to the forest, right? Because then the `random_state` would have been altered by building the tree? (or I'm tired)

_Originally posted by @amueller in https://github.com/scikit-learn/scikit-learn/issues/4783#issuecomment-236036188_

I think this is more generally a concern, because even if a constant seed was passed to the RandomForest initializer, then BaseForest.fit(..) implements 
`random_state = check_random_state(self.random_state)`
which makes ``random_state`` a RandomState() object, followed by (a few lines below):
  ` trees = [self._make_estimator(append=False, random_state=random_state) for i in range(n_more_estimators)]`
All trees appear to share the same RandomState() instance, which is then used to generate indices. Calling ``_generate_sample_indices(...)`` again as part of ``_generate_unsampled_indices(..)`` would seem to _always_ generate different indices. 

Am I missing something, or can this procedure _never_ work correctly?
user: simontindemans
issue_19630: state: open
title: DEP start to raise warning with inconsistent combination of hyperparameters in SVM
body: References #19614.

#### What does this implement/fix? Explain your changes.

Gives a `FutureWarning` when a user configures a `LibSVM` learner with 'linear' `kernel` but also non-default `gamma`.

```python
from sklearn.datasets import load_iris
from sklearn.svm import SVC

x, y = load_iris(return_X_y=True)
clf = SVC(kernel='linear', gamma=1e-6)
clf.fit(x, y)
print(clf.score(x, y))
```
previous output:
```
0.9933333333333333
```
current output:
```
...\scikit-learn\sklearn\svm\_base.py:220: FutureWarning: Setting 'gamma' when using 'linear' kernel may raise a `ValueError` starting in version 1.1 (renaming of 0.26).
  warnings.warn(
SVC(gamma=1e-06, kernel='linear')
0.9933333333333333
```

#### Any other comments?

Went with a `FutureWarning` instead of the `DeprecationWarning` as discussed in the issue, because:
 - `DeprecationWarning` is ignored by default (unless it's code from `__main__`),
 - It does not remove old functionality (setting `gamma` with `kernel='linear'` had no function).
 
I tried to find guidelines for this in the dev docs, but it either isn't there or I couldn't find it.
Just let me know if I should revert it to `DeprecationWarning`.
Also wasn't sure if a note of this should be added to the docs, since they already describe which parameter affects which kernel.

Tests ran with Windows/Py3.8:

```
pytest sklearn/svm
pytest doc/modules/svm.rst
pytest sklearn/tests/test_common.py -k SVC
```
user: PGijsbers
labels: Needs Decision: Requires decision
issue_19615: state: open
title: [RFC] Improve the Ridge solver speed / convergence tradeoff with better solver defaults
body: I did some follow-up work for the discussions on potential problems for `Ridge(normalize=False)` in #19426 and #17444 in the following gist:

https://gist.github.com/ogrisel/cf7ac8bca6725ec02d4cc8d03ce096a7

This gist compares the results between a minimal implementation of ridge with LBFGS  by @agramfort  against scikit-learn's `Ridge(normalize=False, solver="cholesky")`, `Ridge(normalize=False, solver="lsqr")`, `Ridge(normalize=False, solver="svd")` and other solvers.

The main conclusion is that those models seems to converge to the same solution, if we set a low value of `tol` and if we are not too strict on the tolerance: see how I defined atol in the gist. I cannot get convergence to machine precision (only around 1e-6/1e-7 in float64), but maybe this is expected. I am not so sure: I tried to tweak `pgtol` on top of `factr` but it did not improve.

On large-ish problems, L-BFGS can be significantly faster than our the always accurate svd solver (but not always, depends on the regularization) but never faster than the `lsqr` (LAPACK-based) solver. The cholesky solver (our default for dense data) is sometimes faster or slower than lsqr and L-BFGS (depending on regularization) but always faster than SVD (at least with high regularization). The sag/saga solvers are not necessarily as fast as they could be: I noticed that they do no benefit from multi-threading on a multicore machine, so that might be the reason. I have never seen then run faster than lsqr. sag/saga can be significantly faster than L-BFGS despite using fewer cores on problems with `n_samples=5e3` / `n_features=1e3 `and not too much regularization (alpha=1e-6).

There are still problems with `normalize=True` as reported in #17444 but maybe we don't care so much because we are going to deprecate this option, see #17772 by @maikia that I plan to review soon.

Proposed plan of actions:

- [x]  proceed with the deprecation of `normalize=True` in #17772 and <del>therefore put  #17444 on hold (or less of a priority)</del> actually no, see: #19616.
- [x] improve the ridge tests in scikit-learn by explicitly evaluating the min function for different solvers and maybe using a utility such as the `check_min` of my gist.
   Tight tests (but without checking the minimum explicitly) #22910
- [ ] maybe add the ridge_bfgs function as a reference to compare against in our test suite? Since it does not seem to always converge to a very high quality optimum, I am not so sure of the value.
- [ ] maybe we could conduct a more systematic evaluation of various datasets with different shapes, conditioning and regularization to select  a better default than "cholesky" for dense data? "cholesky" is reputed to not be numerically stable, although I did not observe problems in my gist, maybe because of regularization. `lsqr` seems to be very promising, both in terms of speed and accuracy of the solution (according to my `check_min` thingy).
- [x] I am not sure that our default value of `tol=1e-3` is safe. It's probably very solver dependent, but I doubt that it's a good idea for the fast `lsqr` solver for instance (to be benchmarked if we do the above evaluation). https://github.com/scikit-learn/scikit-learn/pull/24465

Other ideas:

- maybe we should consider adding l-bfgs an option? That would require doing more evaluation on sparse data / ill-conditioned data with various values of alpha. Not sure of the benefit compared to `lsqr` or `cholesky` though: they seem to always be more accurate than l-bfgs and faster (especially `lsqr`). And l-bfgs  can be very slow when alpha is small.
- and maybe add a QR-based solver that is supposed to be [more numerically stable](http://www.math.iit.edu/~fass/477577_Chapter_5.pdf)? or is this what LAPACK is doing internally when we use the `lsqr` solver?
user: ogrisel
labels: module:test-suite: everything related to our tests
issue_19614: state: open
title: Give feedback when `svm.SVC` is configured with kernel hyperparameters for a different kernel
body: During our class we noticed some students incorrectly configure some hyperparameters which are irrelevant to the kernel used, for example setting `gamma` when using a linear kernel. We think it could make sense for scikit-learn to give feedback to the user when non-effective settings are configured.

#### Describe the workflow you want to enable

```python
from sklearn.datasets import load_iris
from sklearn.svm import SVC

x, y = load_iris(return_X_y=True)
clf = SVC(kernel='linear', gamma=1e-6)
clf.fit(x, y)
print(clf.score(x, y))
```
current output:
```
0.9933333333333333
```
proposed output, something similar to:
```
UserWarning: Gamma is set but not used because a linear kernel is configured.
0.9933333333333333
```


user: PGijsbers
issue_19610: state: open
title: Archetypal Analysis
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable

Archetypal analysis is similar to clustering analysis. However, instead of cluster centers, it seeks extremal points in the dataset (called "archetypes"), sometimes providing more interpretable results than clustering.

#### Describe your proposed solution

Implement the algorithm proposed in [Cutler and Breiman, 1994](https://digitalassets.lib.berkeley.edu/sdtr/ucb/text/379.pdf). Since it is similar to clustering analysis algorithms, it will fit well within the `fit`, `predict/transform` interface.

In the R ecosystem there is already a specific package for archetypal analysis: [archetypes](https://cran.r-project.org/web/packages/archetypes/index.html)

What do you think about adding this new algorithm? I can open a PR with a proposal :)
user: aleixalcacer
issue_19608: state: open
title: Memory mapping causes disk space usage bloat in big models with many search parameters
body: Hello, 

We use GridSearchCV in a project where our training data is quite large (approx, 20,000 rows x 100 columns/features). We search a large space of hyperparameters, and with 5-10 cross-validation runs total number of model sometimes up to 1000.

When running default parallelization using joblib, the tool generates a lot of memory-mapped disk data and uses a lot of disk space on the scale of hundreds of gigabytes. 

**Solution:**
Once we disable memory mapping, it runs successfully without using the disk. The solution is to call joblib's Parallel(..., max_nbytes=None) within GridSearchCV, see the link to code below. The default for joblib is '1M', which is 1 megabyte. When we pass None, it disables memory mapping.

Suggested improvement is to add a kwarg to GridSearchCV that gets passed to Parallel's max_nbytes.

https://github.com/scikit-learn/scikit-learn/blob/28ee486b44f8e7e6440f3439e7315ba1e6d35e43/sklearn/model_selection/_search.py#L767
user: asansal-quantico
issue_19604: state: open
title: add VCA to Decomposing signals in components (matrix factorization problems)
body: Hi

#### Describe the workflow you want to enable

I would like to use VCA algorithm through sci-kit.
I use a lot this algo to decompose EELS signals in spectromicroscopy. It is fast and robust.
There are already several papers using it in the EELS community.
The seminal paper https://ieeexplore.ieee.org/abstract/document/1411995
has 2236 citations in Google Scholar.

The addition in scikit would enable an easier utilisation in the HyperSpy  package https://hyperspy.org/index.html 

#### Describe your proposed solution

There is an existing Python code
https://github.com/Laadr/VCA

There is an 2nd part to compute the abundance maps 
https://github.com/Laadr/SUNSAL

Thanks for your attention

user: NathalieBrun
issue_19583: state: open
title: Order of features selected in Sequential feature selection transformer
body: Is it currently possible to extract the order in which the features in the sequential feature selection transformer are selected?
Similar to the ranking in the RFE transformer for example. 
user: hakeemo
issue_19573: state: open
title: RobustScaler make the outlier bigger
body: ![image](https://user-images.githubusercontent.com/4510984/109330360-8c58f600-7896-11eb-8e8a-fcdaacc9caf1.png)


#### Expected Results
Remove outlier or better than the raw data.

#### Actual Results
Make the outlier bigger.

> Standardization of a dataset is a common requirement for many machine learning estimators. Typically this is done by removing the mean and scaling to unit variance. **However, outliers can often influence the sample mean / variance in a negative way. In such cases, the median and the interquartile range often give better results.**

#### Versions
![image](https://user-images.githubusercontent.com/4510984/109330259-721f1800-7896-11eb-9c5c-fdb09aa8b2c5.png)


<!-- Thanks for contributing! -->

user: GF-Huang
issue_19562: state: open
title: [MRG] GaussianMixture with BIC/AIC
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Adds the implementation discussed in #19338

#### What does this implement/fix? Explain your changes.
It automatically selects the best GM model based on BIC or AIC among a set of models that are parameterized by:
- Initialization scheme, which could be random, k-means or agglomerative clusterings (as done in `mclust`, see below)
- Covariance constraints
- Number of components

#### Any other comments?
This work is inspired by `mclust` (a popular [package](https://mclust-org.github.io) in R for GM modeling) and its performance is evaluated on synthetic and real data ([Athey, et al., 2020](https://arxiv.org/abs/1909.02688)).

[This notebook](https://nbviewer.jupyter.org/github/tliu68/scikit-learn/blob/gm_ic/Comparing_clustering.ipynb) roughly replicates the clustering experiments in [this tutorial](https://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html) and also includes the `GaussianMixtureIC` algorithm proposed here.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: tliu68
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_19556: state: open
title: FEA Confusion matrix derived metrics
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Continues #17265
Fixes #5516

#### What does this implement/fix? Explain your changes.
Add `metrics.specificity_score` and `metrics.npv_score`, based on `metrics.fpr_tpr_fnr_tnr_scores` from #17265

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Pawel-Kranzberg
issue_19548: state: open
title: Common tests xfailed for Bicluster estimators
body: https://github.com/scikit-learn/scikit-learn/pull/19542 enables bicluster estimators in common test and skips xfail tests that are failing:

- [ ] check_estimators_dtypes
- [ ] check_fit2d_1sample
- [ ] check_fit2d_1feature
- [ ] check_estimator_sparse_data
- [ ] check_methods_subset_invariance
- [ ] check_dont_overwrite_parameters
- [ ] check_fit2d_predict1d

Guidelines for contributors who would like to help fix those issues:

- You can trigger the failure by removing the xfail marker for a specific check and then launch the common tests for that specific check using:
    ```
    pytest -v -k check_estimators_dtypes sklearn/tests/test_common.py
   ```
- Take time to familiarize your-self with [Biclustering algorithms](https://scikit-learn.org/stable/modules/biclustering.html) (seel also [Wikipedia on Biclustering](https://en.wikipedia.org/wiki/Biclustering) to understand how those estimators are specific.
- Feel free to open a small PR for each sub-problem. It's easier to start with a small PR if this is the first time you contribute.
- When opening a PR please share your analysis of the problem in the description of the PR: what is the intention of the check? should it apply to the Bicluster estimators? or should they be considered a legitimate exception to the rule? and if so why?
- If they are an exception, can we use existing [estimators tags](https://scikit-learn.org/stable/developers/develop.html#estimator-tags)? Do we need to introduce a new tag?
- Otherwise there is probably something to change in the code of the Bicluster estimators, in which case don't forget to remove the matching XFAIL marker in your PR.
user: thomasjpfan
labels: Moderate: Anything that requires some knowledge of conventions and best practices
module:test-suite: everything related to our tests
issue_19547: state: open
title: [RFC] Should scalers or other estimators warn when fit on constant features?
body: As discussed in #19527, fitting models on data with constant feature can be surprising.

For instance a `StandardScaler(with_mean=False)` fit on a column with constant values set to `1000.` will let those values passthough unchanged because the variance of the column is null. It can be surprising but is this a problem? Should we warn the user about the presence of such constant features which are typically not predictive for machine learning models?

Which estimator should warn about such constant features? The scalers can naturally detect those because they can detect them when computing the `scale_` attribute. The `QuantileTransformer` could also probably warn about this degenerate case.

`HistGradientBoosting*` and `KBinsDiscretizer` can also do it efficiently when binning the feature values.

If we do so:

- what should be the warning message? Should it be the same for all the models?
- shall we add a standard constructor param to this estimators `constant_feature={'warn', 'drop', 'passthrough', 'zero', 'one'}` with `"warn"` as the default?
- should we generalize this to all estimators? (ogrisel: probably not because it could be expensive and redundant input validation check so we could restrict to the estimators above where it's cheap to check)

Are there legitimate cases where such a warning would be frequent and annoying? For instance `StandardScaler(with_mean=False)` after `OneHotEncoding` with dense output with a categorical feature that has a category that is significantly more frequent than the others in cross-validation loop? A similar problem could happen with after `OrdinalEncoding`. But would `StandardScaler(with_mean=False)` would actually make sense to use in those cases?

user: ogrisel
issue_19533: state: open
title: PolynomialFeatures: allow user defined combinations of features
body: As of v0.24.1, `sklearn.preprocessing.PolynomialFeatures` [has](https://scikit-learn.org/0.24/modules/generated/sklearn.preprocessing.PolynomialFeatures) three options that determine which combinations of features are generated:

1. `degree`: the maximum number of features to combine into a polynomial feature
2. `interaction_only`: filters out any combinations that include the same feature multiple times
3. `include_bias`: adds a column of ones

These are nice options, but are unable to capture every use case for generating polynomial feature combinations. For example, my data has 4 features: a, b, c, d. I want to transform these features into a, ab, ac, ad. I didn't see any way to achieve this will `PolynomialFeatures` directly, so instead I created a subsequent step in my pipeline to select a subset of columns from the `PolynomialFeatures` output.

Letting users specify any combination of features would be a general purpose solution. For example, I propose supporting something like: 

```py
# combinations by feature name (for situations when feature names are available)
PolynomialFeatures(combinations=[("a",), ("a", "b"), ("a", "c"), ("a", "d")])

# combinations by index
PolynomialFeatures(combinations=[(0,), (0, 1), (0, 2), (0, 3)])

# another set of combinations by index that isn't currently possible
PolynomialFeatures(combinations=[(0,) (1,), (0, 0), (1, 1, 1), (0, 1)])
```

Does this make sense? Is there some other way of generating custom combinations of polynomial features in a pipeline that I am overlooking?
user: dhimmel
issue_19526: state: open
title: BIC and AIC scores in _bayesian_mixture.py
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes #19523 


#### What does this implement/fix? Explain your changes.
Added the possibility to get BIC and AIC scores for the Bayesian Gaussian Mixture class. Worked on `sklearn/mixture/_bayesian_mixture.py` and `sklearn/mixture/tests/test_bayesian_mixture.py`.


#### Any other comments?
This is based on how the BIC and AIC scores are calculated for the Gaussian Mixture class in `sklearn/mixture/_gaussian_mixture.py` with slight modifications. Unittests added in `sklearn/mixture/tests/test_bayesian_mixture.py`, based on the unittests for the corresponding functions in `sklearn/mixture/tests/test_gaussian_mixture.py`. 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thoranna
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_19523: state: open
title: BIC and AIC scores for the Bayesian Gaussian Mixture class
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
The [sklearn.mixture.GaussianMixture](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html#sklearn.mixture.GaussianMixture) has built in functions to get the Akaike Information Criterion (AIK) and Bayesian Information Criterion (BIC) scores. However, the [sklearn.mixture.BayesianGaussianMixture](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.BayesianGaussianMixture.html#sklearn.mixture.BayesianGaussianMixture) class does not. 

#### Describe your proposed solution
With some modifications to the functions for Gaussian Mixture, it would be possible to implement BIC and AIC scores for the Bayesian Gaussian Mixture class as well. 

#### Describe alternatives you've considered, if relevant

#### Additional context
I encountered this when working on a ML project where I wanted to use the BIC and AIC scores to evaluate my model when fed different combinations of data. I wanted to use the fact that with the Bayesian Gaussian Mixture class, the effective number of components can be inferred from the data, to speed up my computations.

I'm wondering if this is something that should be incorporated? If there is interest, I made a pull request to solve this. 

Thanks in advance for your time! 

user: thoranna
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_19514: state: open
title: Integrate the TruncatedSVD sign indeterminacy solution
body: #### Describe the workflow you want to enable

[TruncatedSVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html) has the sign indeterminacy problem. However, the sign flip algorithm solves it, and it's already implemented in Scikit-learn. TruncatedSVD uses [randomized_svd](https://scikit-learn.org/stable/modules/generated/sklearn.utils.extmath.randomized_svd.html) for `'randomized'` solver. The `randomized_svd()` has `flip_sign` argument (with default `True` value) with the algorithm for sign determination already implemented.

#### Describe your proposed solution

The only work required to solve sign indeterminacy in `TruncatedSVD` is propagating the `flip_sign` option, adding it to `TruncatedSVD` and passing it to `randomized_svd()`. Of course this applies only when the solver selected is `'randomized'`.

#### Describe alternatives you've considered, if relevant

#### Additional context

user: j-adamczyk
issue_19505: state: open
title: sklearn.manifold.locally_linear_embedding() enforces use of euclidean distance when passing a NearestNeighbors instance with a precomputed distance matrix as parameter
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
<!--
A clear and concise description of what the bug is.
-->
When calling the `locally_linear_embedding()` method with a _NearestNeighbors_ instance as its `X` parameter, the euclidean distance is always used in the standard version of the algorithm (`method = 'standard'`) defeating the purpose of passing down a _NearestNeighbors_ instance with a precomputed distance matrix (`metric = 'precomputed'`).

#### Steps/Code to Reproduce
<!--
Please add a minimal example that we can reproduce the error by running the
code. Be as succinct as possible, do not depend on external data. In short, we
are going to copy-paste your code and we expect to get the same
result as you.

Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
docs = ["Help I have a bug" for i in range(1000)]
vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)
lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

```python
from sklearn.neighbors import NearestNeighbors
from sklearn.manifold import locally_linear_embedding

D = distance_matrix(data) # Precomputation of the distance matrix of some data with a custom distance measure

nbrs = NearestNeighbors(n_neighbors = 5,
                        metric = 'precomputed')
nbrs.fit(D) # NearestNeighbors instance with a precomputed distance matrix

Z,_ =  locally_linear_embedding(nbrs,
                                n_neighbors = 5,
                                n_components = 2) # Passing down the NearestNeighbors instance with a precomputed distance matrix
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
Embeddings computed using the distance matrix passed down to `locally_linear_embedding()` through the _NearestNeighbors_ instance.

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
Embeddings are instead computed using the distance matrix passed down to `locally_linear_embedding()` through the _NearestNeighbors_ instance treating it as a dataset with shape _(n_samples, n_features)_. This happens when calling `barycenter_kneighbors_graph()` on line 320 which in turn calls the `kneighbors()` with a new instance of _NearestNeighbors_. This new instance has attribute `self.effective_metric_='euclidian'`. The problem happens when `kneighbors()` does a internal check on `self.effective_metric_` and decides to use euclidian metric.
#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
import imblearn; print("Imbalanced-Learn", imblearn.__version__)
-->

System:
    python: 3.8.3 (default, Jul  2 2020, 17:30:36) [MSC v.1916 64 bit (AMD64)]
executable: C:\ProgramData\Anaconda3\python.exe
   machine: Windows-10-10.0.19041-SP0

Python dependencies:
          pip: 20.1.1
   setuptools: 49.2.0.post20200714
      sklearn: 0.23.1
        numpy: 1.18.5
        scipy: 1.5.0
       Cython: 0.29.21
       pandas: 1.0.5
   matplotlib: 3.2.2
       joblib: 0.16.0
threadpoolctl: 2.1.0

Built with OpenMP: True

<!-- Thanks for contributing! -->

user: taha-yassine
issue_19498: state: open
title: [MRG] Fix Improvement in RANSAC: is_data_valid should receive subset_idxs 
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes https://github.com/scikit-learn/scikit-learn/issues/19495

#### What does this implement/fix? Explain your changes.


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: alex90r7
issue_19495: state: open
title: Improvement in RANSAC: is_data_valid should receive subset_idxs
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
In some applications of RANSAC the user needs to reject certain iterations based on the indices that were selected during sampling. For example: https://ieeexplore.ieee.org/abstract/document/6856449/, page 3, Section II, C, the authors reject samples that originate only from one sensor. 

#### Describe your proposed solution
The method is_data_valid should get the subset_idxs, this way the user would be able to write custom method that could reject certain cases. 
#### Describe alternatives you've considered, if relevant

#### Additional context

user: alex90r7
issue_19479: state: open
title: Memory errors with Kernel PCA: Incremental Kernel PCA?
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
<!--
A clear and concise description of what the bug is.
-->
Hi,
I am trying to carry a Kernel PCA on a very large dataset (80 000 points). Using the `sklearn.decomposition.KernelPCA` function, this leads to a memory error, as the algorithm would need to create a 80 000 x 80 000 matrix.
The `IncrementalPCA` function seems to be able to solve that kind of problem and to deal with large datasets, but it is not implemented to use with kernels. 

Is there a way to carry a KernelPCA with a large dataset?

Thank you.




#### Versions
System:
    python: 3.9.1 | packaged by conda-forge | (default, Jan 10 2021, 02:55:42)  [GCC 9.3.0]
executable: /storage/mathilde/conda_bak/envs/Scikit/bin/python
   machine: Linux-3.10.0-1062.9.1.el7.x86_64-x86_64-with-glibc2.17

Python dependencies:
          pip: 20.3.3
   setuptools: 49.6.0.post20210108
      sklearn: 0.24.0
        numpy: 1.19.5
        scipy: 1.6.0
       Cython: None
       pandas: 1.2.0
   matplotlib: 3.3.2
       joblib: 1.0.0
threadpoolctl: 2.1.0

Built with OpenMP: True
user: mathildejutras
issue_19449: state: open
title: Wouldn't it be great if fit() from GaussianNB() also accepted 'X' as sparse matrix?
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
For now, `fit()` method from `GaussianNB()` does not accept sparse matrix for `'X'`. If passed, it gives the warning to convert it into a dense array using the `.toarray()` method.
#### Describe your proposed solution
the `fit()` method should accept sparse matrix like `fit()` from `MultinomialNB()`
#### Describe alternatives you've considered, if relevant

#### Additional context

user: vmdhhh
issue_19448: state: open
title: Investigate and document pitfalls with permutation importance on imbalanced classification problems with non-independent features
body: While working on the review of #19411, @glemaitre found out that the permutation importance ranking for different metrics (accuracy, balanced accuracy, precision, recall) could vary a lot on imbalanced classification problems.

Perhaps more surprisingly, it is possible to train models with some features that have very significantly negative importance for some features: the permutation of those features makes their metric improve significantly! Furthermore the same features can be ranked most important for the same model using another choice of metric (e.g. balanced accuracy instead of accuracy or the converse).

This behavior makes it really problematic to rely on permutation importance to inspect models on imbalanced classification problems (which are very common in practice). I think we should really warn our users about this.

It's not that easy to reproduce this phenomenon on minimal reproduction cases but it seems to happen in practice (for instance by rebalancing Adult Census to a 1/10 positive to negative class ratio). 

We still need to clean-up @glemaitre's notebook to properly demonstrate this problem but I wanted to open this issue now to avoid forgetting about this problem.
user: ogrisel
issue_19447: state: open
title: SVC's predict_proba(...) predicts the exact same probability for wildly different inputs
body: I'm using `scikit-learn` version 0.24.1.

I suspect some form of memoization or extreme low sensitivity happening in the SVC `predict_proba(...)` implementation whereby once a SVC model is built from a Pipeline with inputs scaling preprocessing and calibration and when setting the `random_state` argument; I get the exact same predicted probability for wildly different inputs.

I also checked the `decision_function(..)` result and it returns the exact same value for two wildly different `x_test` inputs.

This issue only happens when setting the argument `random_state` e.g. to zero. It is hard to create a MRE here also because of NDA and confidentiality of the dataset and work but I propose to export the pipeline like this:

```
from joblib import dump, load
dump(model, '/some/where/model.joblib')
```
and provide the two wildly different inputs leading to the same exact `predict_proba(...)` and `decision_function(...)` results. Would that be a viable solution to reproduce and fix the possible bug?

A simplified relevant example of my pipeline is the following:
```
pipeline = Pipeline(steps=[('preprocess', MaxAbsScaler()),
                           ('svm', SVC(kernel='rbf', probability=True, 
                                       random_state=0, decision_function_shape='ovr', 
                                       break_ties=False, ...))]
params = [{...}]
model = GridSearchCV(pipeline, params, ...).fit(x_train, y_train).best_estimator_

# and now I get
prob1 = model.predict_proba(x_test1)
prob2 = model.predict_proba(x_test2_wildly_diff_from_test1)
assert np.abs(prob1 - prob2) < 1e-10
```

For example, using `random_seed=0`:
```
>>> import numpy as np
>>> from scipy.spatial import distance
# how far are the two x_test vectors from each other?
>>> distance.cosine(x_test1, x_test2)  # EDIT: very far apart angle-wise e.g. 90°
1.0280449512858494
>>> distance.euclidean(x_test1, x_test2) # very far in euclidean distance
30675.221284568033
>>> model.predict_proba(x_test1)
array([[0.86879653, 0.13120347]])
>>> model.predict_proba(x_test2)
array([[0.86879653, 0.13120347]])
>>> model.decision_function(x_test1)
array([-0.03474242])
>>> model.decision_function(x_test2)
array([-0.03474242])
```

UPDATE: what bothers me is not that they are close, what spooks me is that the two completely different vectors end up getting the exact same distance (to the 1e-10 decimal place) from the decision boundary `decision_function(...)` and therefore the exact probability too `predict_proba(...)`. I'm still thinking how to further validate and scrutinize this case ...

user: bravegag
issue_19433: state: open
title: KBinsDiscretizer produces wrong bins with repeated small values
body: The `KBinsDiscretizer` will fail to produce quantile discretizations if the input data has most of its entries having the same value and corresponding to the first bin:

```python
import numpy as np
from sklearn.preprocessing import KBinsDiscretizer
kb = KBinsDiscretizer(n_bins=3, encode="ordinal", strategy="quantile")
a = np.array([0,0,0,0,0,0,0,0,1,2]).reshape((-1,1))
kb.fit(a).transform(a)
```
```
/home/david/anaconda3/lib/python3.7/site-packages/sklearn/preprocessing/_discretization.py:222: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 0 are removed. Consider decreasing the number of bins.
  'decreasing the number of bins.' % jj)
```
```
array([[0.],
       [0.],
       [0.],
       [0.],
       [0.],
       [0.],
       [0.],
       [0.],
       [0.],
       [0.]])
```

In this case, there's no reason why the last two elements could not be assigned to the highest bucket instead of being grouped with the rest.
user: david-cortes
labels: Needs Decision - API: Requires decision regarding API
issue_19427: state: open
title: Fixed bug handling multi-class classification
body: #### Reference Issues/PRs
#19270 

#### What does this implement/fix? Explain your changes.
Fixes a bug in the function handling multi-class classification by reducing the fixed value of the ovr_decision_function, which is too large.
user: isaack-njama
labels: Needs Decision - Backward Compatibility: Requires decision regarding backward compatibility
issue_19416: state: open
title: KBinsDiscretizer creates wrong bins
body: #### Describe the bug

when binning many identical values, `KBinsDiscretizer` fails to create the appropriate number of bins, complaining
> UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 0 are removed.


#### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.preprocessing import KBinsDiscretizer
X = np.array([0,0,0,0,0,1,1,1,2,3]).reshape((-1,1))
bd = KBinsDiscretizer(n_bins=3, strategy="quantile", encode="ordinal")
bd.fit_transform(X).T
```

#### Expected Results

1. No errors or warnings.
1. The last expression should evaluate to `np.array([[0., 0., 0., 0., 0., 1., 1., 1., 2., 2.]])`

#### Actual Results

1. Warning:
> sklearn/preprocessing/_discretization.py:220: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 0 are removed. Consider decreasing the number of bins.
2. The last expression evaluates to `np.array([[0., 0., 0., 0., 0., 1., 1., 1., 1., 1.]])`

#### Versions

```
System:
    python: 3.9.1 (default, Feb  3 2021, 07:04:15)  [Clang 12.0.0 (clang-1200.0.32.29)]
executable: /.../.virtualenvs/.../bin/python3
   machine: macOS-10.15.7-x86_64-i386-64bit

Python dependencies:
          pip: 21.0.1
   setuptools: 53.0.0
      sklearn: 0.24.1
        numpy: 1.20.1
        scipy: 1.6.0
       Cython: None
       pandas: 1.2.1
   matplotlib: 3.3.4
       joblib: 1.0.0
threadpoolctl: 2.1.0

Built with OpenMP: True
```

PS. could be related to #18638
user: sam-s
labels: Needs Decision - API: Requires decision regarding API
issue_19410: state: open
title: Improvement of PDP and ICE plotting
body: Some discussions from a workshop on interpretation lead us to propose some potential improvements regarding the PDP and ICE plotting utilities.

Regarding PDP, it could be nice to get information regarding the distribution of the in-domain samples for a specific feature. Zhao and Hastie illustrate this in-domain values with a scatter plot in the [following paper](https://web.stanford.edu/~hastie/Papers/pdp_zhao.pdf) (p. 7). We potentially could provide such visualization or an alternative representation that could encode this information (e.g., histogram or box-plot). We should probably make some proposal and see what is best.

Regarding ICE, we currently offer the option `subsample` that will randomly subsample the available ICE. However, it seems to be more meaningful to build percentile from the available ICE lines instead of plotting the ICE lines. Optionally, we could use `subsample` to compute the different percentile.
user: glemaitre
issue_19392: state: open
title: [MRG] Improved parallel execution of plot_partial_dependence / partial_dependence
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### What does this implement/fix? Explain your changes.
Currently, when specifying `n_jobs` for `plot_partial_dependence`, it is parallelized by concurrently computing the partial dependences *per feature* (by calling the function `partial_dependence`). When only one feature is to be plotted, specifying `n_jobs` will lead to no performance boost at best. (Or more general, when `n_jobs` > number of features).

Apart from the fact that I don't think this is documented anywhere, this can be implemented differently to also provide a speedup when the number of features is smaller than n_jobs.

I moved the parallelization to partial_dependence instead. It divides up the grid for which the values are computed and parallelizes among those different parts of the grid. This has two benefits:

- Specifying `n_jobs` now also leads to a performance boost when `n_jobs` > number of features
- `partial_dependence` itself now has a parameter `n_jobs` and can therefore also be parallelized


#### Benchmarks

Some results of timeit before and after the changes with different parameter settings (sorted by relative speedup, executed on my 4-core MacBook):

|    |   n_features |   n_samples |   n_jobs | kind       |   grid_resolution | timeit_result_main                                                    | timeit_result_new                                                     |   best_main |   best_new |   speedup_seconds |   speedup_relative |
|---:|-------------:|------------:|---------:|:-----------|------------------:|:----------------------------------------------------------------------|:----------------------------------------------------------------------|------------:|-----------:|------------------:|-------------------:|
| 21 |            1 |       10000 |        2 | individual |               200 | 24 s ± 146 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)      | 13.4 s ± 6.96 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |   23.8451   |  13.344    |       10.5011     |         0.440388   |
|  5 |            1 |       10000 |        2 | individual |               100 | 13.1 s ± 3.97 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 7.78 s ± 29 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)     |   13.047    |   7.74754  |        5.2995     |         0.406184   |
|  5 |            1 |       10000 |        2 | individual |               100 | 12.9 s ± 267 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 7.78 s ± 29 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)     |   12.6534   |   7.74754  |        4.90582    |         0.387709   |
|  7 |            4 |      100000 |        2 | individual |                10 | 19.2 s ± 1.03 s per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 13.4 s ± 109 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    |   18.1699   |  13.331    |        4.83897    |         0.266317   |
|  1 |            1 |      100000 |        4 | individual |               100 | 24.6 s ± 388 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 18.3 s ± 119 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    |   24.2554   |  18.1511   |        6.10435    |         0.251669   |
| 28 |            1 |      100000 |        4 | individual |                50 | 14.1 s ± 932 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 10.2 s ± 60.3 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |   13.1666   |  10.1478   |        3.0188     |         0.229277   |
| 17 |            1 |       10000 |        2 | average    |               100 | 1.96 s ± 311 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 1.4 s ± 2.64 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    |    1.65249  |   1.39596  |        0.256534   |         0.15524    |
| 16 |            1 |       10000 |        2 | average    |               100 | 1.96 s ± 311 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 1.45 s ± 41.7 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    1.65249  |   1.40574  |        0.246758   |         0.149325   |
| 15 |            1 |       10000 |        2 | average    |               100 | 1.51 s ± 11 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)     | 1.4 s ± 2.64 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    |    1.50335  |   1.39596  |        0.107393   |         0.0714358  |
|  2 |            4 |       10000 |        4 | average    |                50 | 2.43 s ± 117 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 2.15 s ± 1.56 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    2.31589  |   2.15133  |        0.164566   |         0.0710596  |
|  7 |            4 |      100000 |        2 | individual |                10 | 14.3 s ± 47.7 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 13.4 s ± 109 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    |   14.2779   |  13.331    |        0.94689    |         0.0663187  |
| 14 |            1 |       10000 |        2 | average    |               100 | 1.51 s ± 11 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)     | 1.45 s ± 41.7 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    1.50335  |   1.40574  |        0.097618   |         0.0649335  |
|  2 |            4 |       10000 |        4 | average    |                50 | 2.34 s ± 42.6 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 2.15 s ± 1.56 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    2.29967  |   2.15133  |        0.148345   |         0.064507   |
|  0 |            5 |      100000 |        2 | average    |               200 | 7.56 s ± 209 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 6.98 s ± 81.7 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    7.35367  |   6.89909  |        0.45458    |         0.0618168  |
| 33 |            4 |      100000 |        4 | average    |               100 | 3.35 s ± 8.13 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 3.22 s ± 42.7 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    3.34491  |   3.1804   |        0.164512   |         0.0491827  |
| 30 |            4 |       10000 |        2 | average    |               100 | 4.08 s ± 250 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 3.67 s ± 19.7 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    3.82567  |   3.65248  |        0.173185   |         0.0452691  |
|  3 |            5 |      100000 |        1 | average    |                10 | 691 ms ± 14.5 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 649 ms ± 757 µs per loop (mean ± std. dev. of 2 runs, 1 loop each)    |    0.676198 |   0.648693 |        0.0275046  |         0.0406754  |
|  4 |            4 |      100000 |        2 | individual |               200 | 2min 16s ± 718 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)  | 2min 11s ± 583 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)  |  135.885    | 130.866    |        5.01914    |         0.0369366  |
|  8 |            4 |      100000 |        2 | individual |                10 | 14.3 s ± 47.7 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 14.3 s ± 521 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    |   14.2779   |  13.7621   |        0.515808   |         0.0361265  |
|  4 |            4 |      100000 |        2 | individual |               200 | 2min 16s ± 386 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)  | 2min 11s ± 583 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)  |  135.739    | 130.866    |        4.87275    |         0.035898   |
| 32 |            4 |      100000 |        1 | average    |                10 | 556 ms ± 5.98 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 534 ms ± 3.14 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    0.549602 |   0.531212 |        0.0183903  |         0.0334611  |
| 23 |            4 |       10000 |        1 | average    |                50 | 1.85 s ± 353 µs per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 1.81 s ± 1.64 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    1.84987  |   1.80488  |        0.0449915  |         0.0243215  |
| 13 |            5 |       10000 |        2 | individual |               200 | 1min 15s ± 116 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)  | 1min 13s ± 48.3 ms per loop (mean ± std. dev. of 2 runs, 1 loop each) |   75.5593   |  73.7815   |        1.77786    |         0.0235293  |
|  1 |            4 |       10000 |        2 | individual |                50 | 19.6 s ± 1.22 s per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 18 s ± 71.3 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)     |   18.3876   |  17.9554   |        0.432212   |         0.0235056  |
|  1 |            4 |       10000 |        2 | individual |                50 | 18.3 s ± 10.1 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 18 s ± 71.3 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)     |   18.2911   |  17.9554   |        0.335784   |         0.0183577  |
|  0 |            5 |      100000 |        2 | average    |               200 | 7.24 s ± 259 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 6.98 s ± 81.7 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    6.98345  |   6.89909  |        0.0843604  |         0.0120801  |
|  8 |            4 |       10000 |        1 | average    |                10 | 473 ms ± 20.4 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 448 ms ± 311 µs per loop (mean ± std. dev. of 2 runs, 1 loop each)    |    0.452602 |   0.447662 |        0.00493988 |         0.0109144  |
| 24 |            2 |      100000 |        2 | average    |               100 | 2 s ± 13.8 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)      | 1.99 s ± 20.8 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    1.98503  |   1.97201  |        0.013015   |         0.0065566  |
|  3 |            5 |      100000 |        1 | average    |                10 | 653 ms ± 3.02 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 649 ms ± 757 µs per loop (mean ± std. dev. of 2 runs, 1 loop each)    |    0.650019 |   0.648693 |        0.0013255  |         0.00203917 |
| 12 |            4 |       10000 |        1 | individual |               200 | 1min 33s ± 160 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)  | 1min 33s ± 77.4 ms per loop (mean ± std. dev. of 2 runs, 1 loop each) |   93.8372   |  93.8985   |       -0.0613526  |        -0.00065382 |
|  6 |            5 |       10000 |        1 | individual |               200 | 1min 57s ± 36.2 ms per loop (mean ± std. dev. of 2 runs, 1 loop each) | 1min 57s ± 105 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)  |  117.526    | 117.677    |       -0.151023   |        -0.00128502 |
| 19 |            5 |       10000 |        4 | average    |                10 | 1.83 s ± 168 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 1.71 s ± 43.7 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    1.66691  |   1.67113  |       -0.00421947 |        -0.00253131 |
| 18 |            2 |      100000 |        2 | average    |                50 | 1.49 s ± 168 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 1.38 s ± 47.7 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    1.32687  |   1.33121  |       -0.00433733 |        -0.00326884 |
| 20 |            2 |       10000 |        2 | individual |                10 | 4.94 s ± 17.2 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 5.05 s ± 86.7 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   |    4.92447  |   4.96486  |       -0.040398   |        -0.00820353 |
| 22 |            5 |      100000 |        2 | individual |               100 | 1min 35s ± 9.64 s per loop (mean ± std. dev. of 2 runs, 1 loop each)  | 1min 28s ± 961 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)  |   86.3375   |  87.1939   |       -0.856435   |        -0.00991961 |
| 27 |            1 |       10000 |        1 | individual |               100 | 12.3 s ± 2.39 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 12.6 s ± 100 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    |   12.3309   |  12.4598   |       -0.1289     |        -0.0104534  |
| 26 |            2 |       10000 |        2 | individual |               100 | 14.8 s ± 18.8 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 15 s ± 5.16 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)     |   14.8146   |  14.984    |       -0.169479   |        -0.01144    |
|  9 |            4 |      100000 |        2 | individual |                10 | 13.1 s ± 35.9 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 13.4 s ± 109 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    |   13.058    |  13.331    |       -0.272955   |        -0.0209032  |
|  9 |            4 |       10000 |        1 | individual |               200 | 1min 32s ± 405 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)  | 1min 33s ± 77.4 ms per loop (mean ± std. dev. of 2 runs, 1 loop each) |   91.6971   |  93.8985   |       -2.20142    |        -0.0240075  |
|  6 |            5 |       10000 |        1 | individual |               200 | 1min 56s ± 1.71 s per loop (mean ± std. dev. of 2 runs, 1 loop each)  | 1min 57s ± 105 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)  |  114.774    | 117.677    |       -2.9032     |        -0.025295   |
| 25 |            2 |       10000 |        2 | individual |               200 | 25.5 s ± 181 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 26 s ± 2.47 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)     |   25.3149   |  25.9944   |       -0.679459   |        -0.0268402  |
| 10 |            4 |      100000 |        2 | individual |                10 | 13.1 s ± 35.9 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 14.3 s ± 521 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    |   13.058    |  13.7621   |       -0.704036   |        -0.053916   |
| 11 |            4 |       10000 |        1 | average    |                10 | 427 ms ± 2.34 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)   | 448 ms ± 311 µs per loop (mean ± std. dev. of 2 runs, 1 loop each)    |    0.42461  |   0.447662 |       -0.0230521  |        -0.05429    |
| 29 |            2 |       10000 |        4 | individual |                10 | 6.24 s ± 815 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    | 6.28 s ± 187 ms per loop (mean ± std. dev. of 2 runs, 1 loop each)    |    5.42025  |   6.09757  |       -0.677328   |        -0.124963   |

So the modification seems to lead to a speedup in cases when `effective_n_jobs > n_features` and to no significant change in most other cases.

The code I used to come up with those benchmarks:

```
from sklearn.inspection import plot_partial_dependence
from sklearn.inspection import partial_dependence
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import make_regression

import pandas as pd
from matplotlib import pyplot
import random


timing_results = []

def timeit_and_save_result(X, **kwargs):
    print(f"Executing timeit with {kwargs}")
    n_features = kwargs.pop('n_features')
    n_samples = kwargs.pop('n_samples')
    
    pyplot.close("all") 
    timeit_res = %timeit -r 2 -o plot_partial_dependence(est, X, range(n_features), random_state=0, **kwargs)
    
    timing_results.append({
        'timeit_result': timeit_res,
        'n_features': n_features,
        'n_samples': n_samples,
        **kwargs
    })
    

X, y = make_regression(n_samples=int(1e5), n_features=10, random_state=0)

est = RandomForestRegressor(n_estimators=10, n_jobs=4, random_state=0)
est.fit(X, y)


param_grid = {
    'n_samples': [int(1e4), int(1e5)],
    'n_features': [1, 2, 4, 5],
    'n_jobs': [1, 2, 4],
    'kind': ['average', 'individual'],
    'grid_resolution': [10, 50, 100, 200]
}


def sample_params(grid):
    return {k: random.choice(v) for k,v in grid.items()}


for _ in range(30):
    params = sample_params(param_grid)
    X_sample = X[:params['n_samples']]
    timeit_and_save_result(X_sample, **params)
``` 
user: timo-stoettner
issue_19355: state: open
title: RFC Unified way to process batches in the "MiniBatch" estimators
body: Currently we have 3 "MiniBatch" estimators: `MiniBatchKMeans`, `MiniBatchDictionaryLearning` and `MiniBatchSparsePCA`. The sparse pca is essentially the same as dict learning with dict and code exchanged. A new one is also incoming.

Since `MiniBatchKMeans` and `MiniBatchDictionaryLearning` are being reworked (#17622 and #18975), and `MiniBatchNMF` is on development (#16948), I think it's a good moment to unify the way we process the minibatches, because they currently all implement a different way :S 

Here are the existing implementations with their pros and cons:
- Sample a batch from the data at each iteration (with or without replacement) (currently in minibatch kmeans)
  ```
  for i in range(n_iter):
       this_X = X[np.random.choice(n_samples, size=batch_size, replace=True/False)]
  ```
   - pros: does not require the data to be shuffled; all batches have the same size
   - cons: it's not cheap, fancy indexing makes a copy of the batch. Not clear epochs if batch_size does not divide n_samples.
               no guarantee that all samples are seen at each epoch.

- Pre-define the batches by generating contiguous slices and cycle (currently in minibatch dict-learning)
  ```
  batches = gen_batches(n_samples, batch_size)
  batches = itertools.cycle(batches)
  
  for i, batch in zip(range(n_iter), batches):
      this_X = X[batch]
  ```
  - pros: clear epochs (every completed cycle). free, batches are views on on data.
  - cons: The last batch of each epoch can be smaller that batch_size (which can mess a bit with the stopping criterion). Requires the data to be shuffled (hence the shuffle param). The data is not re-shuffled between each epoch so there's still some pattern in the batches.


@scikit-learn/core-devs please give your opinion on your preferred solution. Maybe there's a most commonly accepted way of doing that / best practices ? Feel free to propose alternatives as well ! Also feel free to edit the pros and cons if I forgot something

user: jeremiedbb
issue_19352: state: open
title: Interactive Imputer cannot accept PLSRegression() as an estimator due to "shape mismatch"
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
When setting the estimator as PLSRegression(), a ValueError is triggered by module '_iteractive.py' in line 348, caused by "shape mismatch"

#### Steps/Code to Reproduce

Example:
```python
import numpy as np

from sklearn.datasets import fetch_california_housing
from sklearn.cross_decomposition import PLSRegression
from sklearn.experimental import enable_iterative_imputer  # noqa
from sklearn.impute import IterativeImputer

rng = np.random.RandomState(42)

X_california, y_california = fetch_california_housing(return_X_y=True)
X_california = X_california[:400]
y_california = y_california[:400]

def add_missing_values(X_full, y_full):
    n_samples, n_features = X_full.shape

    # Add missing values in 75% of the lines
    missing_rate = 0.75
    n_missing_samples = int(n_samples * missing_rate)

    missing_samples = np.zeros(n_samples, dtype=bool)
    missing_samples[: n_missing_samples] = True

    rng.shuffle(missing_samples)
    missing_features = rng.randint(0, n_features, n_missing_samples)
    X_missing = X_full.copy()
    X_missing[missing_samples, missing_features] = np.nan
    y_missing = y_full.copy()

    return X_missing, y_missing

X_miss_california, y_miss_california = add_missing_values(
    X_california, y_california)

imputer = IterativeImputer(estimator=PLSRegression(n_components=2))

X_imputed = imputer.fit_transform(X_miss_california)
print(X_imputed)
```


#### Expected Results: after applying the workaround below:
```python
[[   8.3252       41.            6.98412698 ...    2.55555556
    37.88       -122.25930206]
 [   8.3014       21.            6.23813708 ...    2.10984183
    37.86       -122.22      ]
 [   7.2574       52.            8.28813559 ...    2.80225989
    37.85       -122.24      ]
 ...
 [   3.60438721   50.            5.33480176 ...    2.30396476
    37.88       -122.29      ]
 [   5.1675       52.            6.39869281 ...    2.44444444
    37.89       -122.29      ]
 [   5.1696       52.            6.11590296 ...    2.70619946
    37.8709526  -122.29      ]]
```

#### Actual Results
```python
File "/home/hushsh/py3/lib/python3.6/site-packages/sklearn/impute/_iterative.py", line 348, in _impute_one_feature
    X_filled[missing_row_mask, feat_idx] = imputed_values
ValueError: shape mismatch: value array of shape (27,1) could not be broadcast to indexing result of shape (27,) 
```

#### Versions
System:
    python: 3.6.9 (default, Oct  8 2020, 12:12:24)  [GCC 8.4.0]
executable: /home/hushsh/raid_data/py3/bin/python
   machine: Linux-5.4.0-60-generic-x86_64-with-LinuxMint-19.3-tricia

Python dependencies:
          pip: 21.0.1
   setuptools: 47.3.1
      sklearn: 0.24.1
        numpy: 1.18.1
        scipy: 1.4.1
       Cython: 0.29.15
       pandas: 1.1.3
   matplotlib: 3.1.2
       joblib: 0.14.1
threadpoolctl: 2.1.0

Built with OpenMP: True

#### My Workaround that fixed the bug: Insert the following three lines before line 348
```python
shape_imputed_values = imputed_values.shape
if len(shape_imputed_values)>1:
    # convert 2D array to 1D array fixes the bug:
    imputed_values = imputed_values.reshape(shape_imputed_values[0])
```

<!-- Thanks for contributing! -->

user: firshu
labels: Hard: Hard level of difficulty
issue_19343: state: open
title: ExpSineSquared GP Kernel is not PD
body: Here's a bit of test code:

```python
import numpy as np
import scipy
from sklearn.gaussian_process.kernels import ExpSineSquared

L = 1.0

# create some train/test data on a grid
train_len = 4
r = np.linspace(0, L, train_len)
train_x, train_y = np.meshgrid(r, r)
train_in = np.stack((train_x.flatten(), train_y.flatten()), axis=-1)

# get the kernel
kernel = ExpSineSquared()

K = kernel(train_in) + 1e-4 * np.eye(train_len**2)

print(np.sort(np.linalg.eigvals(K)))
```

This rather popular kernel for modeling periodic functions doesn't seem to be a truly positive definite kernel, at least in the sklearn implementation, as some of the eigenvalues printed are negative. Is this an issue with the sklearn implementation, or is this a weird issue with the kernel itself?

I think folks found something similar in #9824 
user: tawe141
issue_19338: state: open
title: Gaussian Mixture with BIC/AIC
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
Clustering with Gaussian mixture modeling frequently entails choosing the best model parameter such as the number of components and covariance constraint. This [demonstration](https://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_selection.html#sphx-glr-auto-examples-mixture-plot-gmm-selection-py)  is very helpful to me but I think it might be great to have a class like `LassoLarsIC` that does the job automatically.

#### Describe your proposed solution
Add a class (say `GaussianMixtureIC`, for example) that automatically selects the best GM model based on BIC or AIC among a set of models. As mentioned above, the set of models would be parameterized by:
- Initialization scheme, which could be random, k-means or agglomerative clusterings (as done in `mclust`, see below)
- Covariance constraints
- Number of components

#### Additional context
`mclust` is a package in R for GM modeling. The [original publication](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=Jeffrey+D.+Banfield+and+Adrian+E.+Raftery.+Model-based+gaussian+and+non-gaussian+clustering.+Biometrics%2C+49%3A803–821%2C+1993.&btnG=) and the most [recent version](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=Luca+Scrucca%2C+Michael+Fop%2C+Thomas+Brendan+Murphy%2C+and+Adrian+E.+Raftery.+mclust+5%3A+clustering%2C+classification+and+density+estimation+using+Gaussian+finite+mixture+models.+The+R+Journal%2C+8%281%29%3A+205–233%2C+2016.&btnG=) have been cited in 2703 and 956 articles, respectively (Banfield & Raftery, 1993; Scrucca et al., 2016). It incorporates different initialization strategies (including agglomerative clusterings) for EM algorithm and enables automatic model selection via BIC for different combinations of clustering options (Scrucca et al., 2016).
user: tliu68
issue_19291: state: open
title: Elapsed time for transforming in Pipeline and FeatureUnion
body: Here https://github.com/scikit-learn/scikit-learn/pull/11364 fitting time logging was implemented (for Pipeline, FeatureUnion). 
And I think it is useful to have logs for transforming time too. 
I (with my collaegues) use workaround for it. Maybe I am not alone who wants it. What do you think?

user: MezentsevIlya
issue_19270: state: open
title: Bug in SVM._ovr_decision_function
body: Dear scikit learn developpers,

We think we have found a bug in _ovr_decision_function (in svm/base.py), handling the case of multi-class classification.

This function sums the votes of each class against all others, and then - to break ties - adds the normalized confidence to the sum of votes.

However, if we got it right, the sum of votes should be normalized to be centered on 0 before returning it (so that values > 0 would correspond with a positive overall vote for this class).
In fact, this centering on 0 is not done. Thus, for example, if there are 4 classes, the sum of votes (before adding confidence info) would be in the range 0-3.

The result is that the values returned from _ovr_decision_function are too large by a fixed amount, which depends on the number of classes -- to be precise, by (n_classes - 1) / 2

We think the solution should be to reduce this fixed value from the return value of _ovr_decision_function.

So the last line of this function, which is currently 

**return votes + transformed_confidences**

Should be:

**return votes + transformed_confidences - (n_classes - 1) / 2**

Thank you
Dror Dotan dror.dotan@gmail.com
Fosca Al Roumi fosca.al.roumi@gmail.com


user: drordotan
labels: Needs Decision - Backward Compatibility: Requires decision regarding backward compatibility
issue_19258: state: open
title: sklearn Birch memory usage is over 3x higher than original BIRCH (+ Python overheads)
body: The original BIRCH algorithm defined clustering features as CF=(N,LS,SS) where N is an integer, LS is a vector ("linear sum") and SS is a scalar value (sum of squared values). Assuming double precision, a clustering feature for d dimensional data should hence use `4+8*d+8` bytes.
Memory efficiency is the key motivation of BIRCH.

The sklearn implementation will require WAY more memory, and its not just to boxing overhead:
https://github.com/scikit-learn/scikit-learn/blob/8c6a045e46abe94e43a971d4f8042728addfd6a7/sklearn/cluster/_birch.py#L286-L290
https://github.com/scikit-learn/scikit-learn/blob/8c6a045e46abe94e43a971d4f8042728addfd6a7/sklearn/cluster/_birch.py#L292-L297
as you can see, it stores (N,LS,LS/N,SS,||LS/N||) instead, about `4+8*d*2+8*2` which is about twice the original memory requirements not even taking boxing overhead into account.

The CFTree in BIRCH is meant to be low level - largely a list of CFs and child pointers for inner nodes. Here, the current sklearn implementation uses substantially more memory, too:
https://github.com/scikit-learn/scikit-learn/blob/8c6a045e46abe94e43a971d4f8042728addfd6a7/sklearn/cluster/_birch.py#L139-L152
We first have several constants replicated: threshold, branching factor, is_leaf and n_features are redundantly stored in every node of the tree.
`subclusters_` is okay, that is what you would expect: a list of `_CFSubcluster`; except that this one is not preallocated in size.
`init_centroids_` is a *copy* of the centroids of the CFs, `init_sq_norm_` of their norms. Then there is a double-chaining of leaves in every node (leaf or not) via `prev_leaf_` and `next_leaf_`, but they are only navigated in forward order ever, and _get_leaves will likely be called only once and it would be possible to collect them by tree traversal more efficiently rather than maintaining this list throughout the tree construction. There even is an additional `dummy_leaf_` to access the double-linked list because of that, which is completely initialized with an empty matrix.
The additional copies of LS/N and ||LS/N|| increase the Birch memory consumption to be more than three times the original memory; not taking any additional Python overheads into account.
user: kno10
issue_19254: state: open
title: [MRG] Feature Proximity Matrix in RandomForest class
body: #### Reference Issues/PRs
Implements  #18789

Largely inspired by @glouppe: https://stackoverflow.com/questions/18703136/proximity-matrix-in-sklearn-ensemble-randomforestclassifier


#### What does this implement/fix? Explain your changes.
This implements the proximity matrix as a method of the RandomForestClassifier class. If you want to see why it's important, please take a look at the following article : https://www.fromthegenesis.com/proximity-matrix-in-random-forest/



user: jdelaunay
issue_19253: state: open
title: TST Add test for numerical issues in BIRCH
body: BIRCH has numerical issues (c.f., A. Lang, E. Schubert: BETULA: Numerically Stable CF-Trees for BIRCH Clustering. SISAP 2020: 281-296, https://arxiv.org/abs/2006.12881).

This pull request adds a unit test that demonstrates how severe the issues can become when the tree is fed input data with float16 or float32, and hence should always use at least float64 precision.
user: kno10
issue_19252: state: open
title: Document differences to the BIRCH algorithm
body: This pull requests adds some of the difference of the current implementation to the BIRCH paper.

Honestly, its fairly far from the original BIRCH in my opinion, and will require substantial work to become closer.
user: kno10
issue_19250: state: open
title: train_test_split stratify with a pandas Int32 dtype causes error
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
<!--
A clear and concise description of what the bug is.
-->

It appears since 0.23.0 using multiple columns in train_test_split's stratify option results in an error if one column is the pandas nullable int `Int32Dtype()` type. Error does not occur in 0.22.2.post1

#### Steps/Code to Reproduce
<!--
Please add a minimal example that we can reproduce the error by running the
code. Be as succinct as possible, do not depend on external data. In short, we
are going to copy-paste your code and we expect to get the same
result as you.

Example:

```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
docs = ["Help I have a bug" for i in range(1000)]
vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)
lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

The following code will work in 0.22.2.post1 but will give an error in 0.23.0 or later (including 0.24.1). If only column `a` or `c` is chosen for `stratify` it will work. It will also work if column `c` is a numpy `int32` type rather than the nullable pandas `Int32`.

```
from sklearn.model_selection import train_test_split
import pandas as pd

df = pd.DataFrame({'a':['a','b','b']*10,'b':list(range(30)),'c':[1,2,2]*10})
df['c']=df['c'].astype('Int32')

x_train, x_test, y_train, y_test=train_test_split(df.drop(columns='a'),df['a'],test_size=.33, stratify=df[['a','c']])

```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
No error is thrown and results are:
```
>>> x_train
   b  c
4  4  2
1  1  2
5  5  2
3  3  1
>>> x_test
   b  c
0  0  1
2  2  2
>>> y_train
4    b
1    b
5    b
3    a
Name: y_train, dtype: object
>>> y_test
0    a
2    b
Name: y_test, dtype: object
```
#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_split.py", line 2197, in train_test_split
    train, test = next(cv.split(X=arrays[0], y=stratify))
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_split.py", line 1793, in split
    y = check_array(y, ensure_2d=False, dtype=None)
  File "/usr/local/lib/python3.7/site-packages/sklearn/utils/validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/sklearn/utils/validation.py", line 560, in check_array
    array = array.astype(dtype)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/generic.py", line 5548, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors,)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 604, in astype
    return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 409, in apply
    applied = getattr(b, f)(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/internals/blocks.py", line 595, in astype
    values = astype_nansafe(vals1d, dtype, copy=True)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/dtypes/cast.py", line 997, in astype_nansafe
    return arr.astype(dtype, copy=True)
ValueError: could not convert string to float: 'a'
```

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
import imblearn; print("Imbalanced-Learn", imblearn.__version__)
-->
System:
    python: 3.7.8 (default, Jul  4 2020, 08:48:22)  [Clang 11.0.0 (clang-1100.0.33.17)]
executable: /usr/local/opt/python/bin/python3.7
   machine: Darwin-18.7.0-x86_64-i386-64bit

Python dependencies:
          pip: 20.0.2
   setuptools: 46.0.0
      sklearn: 0.24.1
        numpy: 1.19.4
        scipy: 1.4.1
       Cython: None
       pandas: 1.1.5
   matplotlib: 3.2.1
       joblib: 0.14.1
threadpoolctl: 2.1.0

Built with OpenMP: True

<!-- Thanks for contributing! -->

user: dhorkel
issue_19230: state: open
title: [openSUSE TUmbleweed] Test failed, test_convergence_dtype_consistency, AssertionError: Not equal to tolerance rtol=0.001, atol=0
body: Hello
I tried to build scikit-learn 0.24.1 on openSUSE Tumbleweed on i586 architecture.
One test failed, could it be due to limitation of 32 bit?
The test pass on x86_64.
Here are the deps installed
```
python38-base-3.8.6-2.1
blas-devel-3.8.0-4.1
python38-appdirs-1.4.4-2.2
python38-attrs-20.3.0-1.2
python38-iniconfig-1.1.1-1.2
python38-more-itertools-8.6.0-1.2
python38-olefile-0.46-2.5
python38-ordered-set-3.1.1-5.2
python38-py-1.9.0-1.4
python38-pybind11-2.5.0-2.3
python38-pyparsing-2.4.7-2.3
python38-threadpoolctl-2.1.0-1.3
python38-toml-0.10.2-1.2
python38-wcwidth-0.2.5-2.2
python38-pytz-2020.5-1.2
python38-six-1.15.0-2.2
python38-kiwisolver-1.3.1-1.2
python38-Cycler-0.10.0-4.5
python38-python-dateutil-2.8.1-1.5
python38-zipp-3.4.0-1.2
python38-packaging-20.4-1.3
python38-psutil-5.7.3-2.2
python38-devel-3.8.6-2.1
python38-numpy-1.19.5-88.1
python38-Pillow-8.0.1-2.2
python38-importlib-metadata-3.3.0-1.2
python38-lz4-3.0.2-2.4
python38-Cython-0.29.21-1.4
python38-numpy-devel-1.19.5-88.1
python38-setuptools-44.1.1-3.2
python38-scipy-1.6.0-47.1
python38-matplotlib-3.3.3-44.3
python38-joblib-0.16.0-6.2
python38-pluggy-0.13.1-1.5
python38-nose-1.3.7-11.11
python38-pytest-6.1.2-3.2
```


```
[ 1588s] ============================= test session starts ==============================
[ 1588s] platform linux -- Python 3.6.12, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3.6
[ 1588s] rootdir: /home/abuild/rpmbuild/BUILDROOT/python-scikit-learn-0.24.1-44.1.i386/usr/lib/python3.6/site-packages/sklearn
[ 1644s] collecting ... collected 18997 items / 16 deselected / 1 skipped / 18980 selected
...
[ 6542s] =================================== FAILURES ===================================
[ 6542s] ______________________ test_convergence_dtype_consistency ______________________
[ 6542s] 
[ 6542s]     def test_convergence_dtype_consistency():
[ 6542s]         # float 64 transformer
[ 6542s]         X_64 = Xdigits[:100].astype(np.float64)
[ 6542s]         rbm_64 = BernoulliRBM(n_components=16, batch_size=5, n_iter=5,
[ 6542s]                               random_state=42)
[ 6542s]         Xt_64 = rbm_64.fit_transform(X_64)
[ 6542s]     
[ 6542s]         # float 32 transformer
[ 6542s]         X_32 = Xdigits[:100].astype(np.float32)
[ 6542s]         rbm_32 = BernoulliRBM(n_components=16, batch_size=5, n_iter=5,
[ 6542s]                               random_state=42)
[ 6542s]         Xt_32 = rbm_32.fit_transform(X_32)
[ 6542s]     
[ 6542s]         # results and attributes should be close enough in 32 bit and 64 bit
[ 6542s]         assert_allclose(Xt_64, Xt_32,
[ 6542s]                         rtol=1e-06, atol=0)
[ 6542s]         assert_allclose(rbm_64.intercept_hidden_, rbm_32.intercept_hidden_,
[ 6542s]                         rtol=1e-06, atol=0)
[ 6542s]         assert_allclose(rbm_64.intercept_visible_, rbm_32.intercept_visible_,
[ 6542s]                         rtol=1e-05, atol=0)
[ 6542s]         assert_allclose(rbm_64.components_, rbm_32.components_,
[ 6542s] >                       rtol=1e-03, atol=0)
[ 6542s] E       AssertionError: 
[ 6542s] E       Not equal to tolerance rtol=0.001, atol=0
[ 6542s] E       
[ 6542s] E       Mismatched elements: 1 / 1024 (0.0977%)
[ 6542s] E       Max absolute difference: 1.64902041e-07
[ 6542s] E       Max relative difference: 0.00120168
[ 6542s] E        x: array([[-0.314464, -0.269352, -0.050074, ...,  0.032314, -0.184839,
[ 6542s] E               -0.34867 ],
[ 6542s] E              [-0.315646, -0.263143, -0.079185, ...,  0.054781, -0.178217,...
[ 6542s] E        y: array([[-0.314464, -0.269352, -0.050074, ...,  0.032314, -0.184839,
[ 6542s] E               -0.34867 ],
[ 6542s] E              [-0.315646, -0.263143, -0.079185, ...,  0.054781, -0.178217,...
[ 6542s] 
[ 6542s] ../../../BUILDROOT/python-scikit-learn-0.24.1-44.1.i386/usr/lib/python3.6/site-packages/sklearn/neural_network/tests/test_rbm.py:232: AssertionError
[ 6542s] =============================== warnings summary ===============================
```

Full log here: https://build.opensuse.org/build/home:andythe_great:branches:devel:languages:python:numeric/openSUSE_Tumbleweed/i586/python-scikit-learn/_log

Thank you.
user: kevinsmia1939
labels: Needs Decision: Requires decision
module:test-suite: everything related to our tests
issue_19224: state: open
title: check_decision_proba_consistency fails with LinearDiscriminantAnalysis
body: The following common check recently started to fail on a Windows CI job:

```python-traceback
name = 'LinearDiscriminantAnalysis'
estimator_orig = LinearDiscriminantAnalysis()

    @ignore_warnings(category=FutureWarning)
    def check_decision_proba_consistency(name, estimator_orig):
        # Check whether an estimator having both decision_function and
        # predict_proba methods has outputs with perfect rank correlation.
    
        centers = [(2, 2), (4, 4)]
        X, y = make_blobs(n_samples=100, random_state=0, n_features=4,
                          centers=centers, cluster_std=1.0, shuffle=True)
        X_test = np.random.randn(20, 2) + 4
        estimator = clone(estimator_orig)
    
        if (hasattr(estimator, "decision_function") and
                hasattr(estimator, "predict_proba")):
    
            estimator.fit(X, y)
            # Since the link function from decision_function() to predict_proba()
            # is sometimes not precise enough (typically expit), we round to the
            # 10th decimal to avoid numerical issues.
            a = estimator.predict_proba(X_test)[:, 1].round(decimals=10)
            b = estimator.decision_function(X_test).round(decimals=10)
>           assert_array_equal(rankdata(a), rankdata(b))
E           AssertionError: 
E           Arrays are not equal
E           
E           Mismatched elements: 2 / 20 (10%)
E           Max absolute difference: 0.5
E           Max relative difference: 0.02631579
E            x: array([ 7. ,  8. , 11. ,  9. , 17. , 10. ,  5. , 14. ,  6. ,  1. , 19.5,
E                   4. ,  2. , 16. , 12. , 13. ,  3. , 15. , 19.5, 18. ])
E            y: array([ 7.,  8., 11.,  9., 17., 10.,  5., 14.,  6.,  1., 20.,  4.,  2.,
E                  16., 12., 13.,  3., 15., 19., 18.])
```

This happened on this PR which should not have any impact on the behavior `LinearDiscriminantAnalysis.predict_proba`: #17743.

https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=25474&view=logs&j=d32b16b6-cb9d-571b-e765-de83708fb8dd&t=b93f76c1-c2c9-579e-c2ec-c4f438af1261


I suspect the test to be too brittle. Maybe using a test set more related to the original distribution (blobs) would avoid ties or caused by arbitrary rounding?
user: ogrisel
labels: module:test-suite: everything related to our tests
issue_19223: state: open
title: [MRG] Support for multiouput in VotingRegressor and VotingClassifier (Fixes #18289)
body: #### Reference Issues/PRs
Fixes #18289

#### What does this implement/fix? Explain your changes.
Adding the support for multiouput in VotingRegressor and VotingClassifier.

#### Any other comments?

This is my first real contribution, so feel free to give your feedback/edit directly my code.
user: DavidQuere
issue_19213: state: open
title: Official citation authors not as up to date as on Arxiv
body: https://scikit-learn.org/stable/about.html#citing-scikit-learn lists the JMLR paper to cite Scikit-learn.
https://arxiv.org/abs/1201.0490 is the same paper but updates the list of authors as at 2018 (but is incessantly out of date). Yet we don't ask anyone to cite it. Should we?
user: jnothman
issue_19201: state: open
title: kernel pca doc additions
body: 
#### Reference Issues/PRs

Answers #17171 

#### What does this implement/fix? Explain your changes.

Added details on the Kernel PCA method documentation to clarify any misunderstandings about the shapes of
the objects manipulated.

#### Any other comments?

I went with `doc/images` folder to add an image but I'm not sure it was the right place to put it. Please check and confirm it's OK. 
user: Asami-1
issue_19190: state: open
title: Enhancement to Confusion Matrix Output Representation for improving readability #19012 
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #19012 

#### What does this implement/fix? Explain your changes.
When you have multiple levels you can have difficulty reading the ndarray, associating the levels with the True and Predicted values. It is an enhancement to the output of confusion matrix function, better representing the true and predicted values for multilevel classes.

- Returns a confusion matrix in dict representation with labels as keys ('true', 'pred')
- Example:
```python
    >>> y_true = ["cat", "ant", "cat", "cat", "ant", "bird"]
    >>> y_pred = ["ant", "ant", "cat", "cat", "ant", "cat"]
    >>> confusion_matrix(y_true, y_pred, labels=["ant", "bird", "cat"], pprint=True)
    {('ant', 'ant'): 2, ('bird', 'ant'): 0, ('cat', 'ant'): 1,
     ('ant', 'bird'): 0, ('bird', 'bird'): 0, ('cat', 'bird'): 0,
     ('ant', 'cat'): 0, ('bird', 'cat'): 1, ('cat', 'cat'): 2}
```
#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: shubhamdo
issue_19187: state: open
title: [WIP] Add sparse matrix support for histgradientboostingclassifier
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md
-->
This pull request adds a crude support of sparse matrix for HistGradientBoostingClassifier.
The handling of the sparse matrix is rather crude to minimize the PR and avoid touching cython for now.

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #15336 


#### What does this implement/fix? Explain your changes.
It implements sparse matrix support for HistGradientBoostingClassifier.
There are two major differences when handling sparse matrix : during fit and during transform.

##### Fit

At fitting time, the sparse matrix can be safely converted to a dense one because only unique values are important.
This is done by getting the non-null values and then appending 0 to the list of values.
The matrix can now be treated the same way a dense one is.

That way, it avoids duplicating the code and having to test a new algorithm.

##### Transform

Transform is more tricky.
The main problem here (as @NicolasHug explained) is that we want the bin with value 0 to be mapped to the value 0.
That way we can take advantage of the sparse structure to treat missing values as 0.
In the returned matrix, values in the bins [0, actual_bin_zero) are right-shifted by one and the bin actual_bin_zero is mapped to 0.
The transform function returns actual_bin_zero alongside the matrix.

The second short-term problem is that the core of the binning logic is done in _map_to_bins which is a cython function.
I prefer not having to touch to the cython code for now so I worked around it.

The process is the following :
1. We pass a 1x1 zero-filled matrix to _map_to_bins to determine the actual_bin_zero.
2. We pack the sparse matrix into a dense one (process described in details bellow)
3. We pass the packed matrix to _map_to_bins to map the bins.
4. We shift the mapped bins.
5. We unpack the dense matrix to a sparse one, conserving the original structure.

##### Packing / Unpacking

To allow _map_to_bins to "process" a sparse matrix, we pack it into a dense one.
This is done column wise and the resulting matrix is the smallest possible that can fit the data.
The function returns the packed matrix and the row indices of the former values.

Unpacking the matrix is the exact opposite : we use the row indices to put the values into the correct position in the sparse matrix.

Example :
```python
>>> a = csr_matrix(array(
      [[   1, 1000,    0],
       [   0,  999,   -1],
       [   2,    0,   -2],
       [   3,    0,   -3],
       [   0,    0,   -4]]))
>>> _pack_matrix(a)
(
array([[   1, 1000,   -1],
       [   2,  999,   -2],
       [   3,    0,   -3],
       [   0,    0,   -4]]), 
array([[ 0.,  0.,  1.],
       [ 2.,  1.,  2.],
       [ 3., nan,  3.],
       [nan, nan,  4.]])
)
```

#### Any other comments?

The main performance bottleneck is the packing / unpacking of the matrix.
I kept that fact in mind and always used the sparsity of the matrix to my advantage.

In the future, the cython code need to be retouched to make place to a cleaner flow.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: StealthyKamereon
issue_19181: state: open
title: Create plot_contigency_matrix.py
body: Add plot_contingency_matrix / ContingencyMatrixDisplay

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->#19107


#### What does this implement/fix? Explain your changes.
Feature adding to visualize contingency matrix

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: yash-clear
issue_19178: state: open
title: Nystroem has misleading parts makes precomputed kernel seemingly possible.
body: #### Describe the issue linked to the documentation
To my understanding the sklearn.kernel_approximation.Nystroem class is currently not supporting 'precomputed' kernel correctly. And it is true that in the API that there is no option stated for the kernel attribute, but in the _get_kernel_params method, you can see an if statement including self.kernel!='precomputed', which made lead the reader think that it's actually supporting precomputed kernel.  

Also in https://github.com/scikit-learn/scikit-learn/blob/42aff4e2edd8e8887478f6ff1628f27de97be6a3/sklearn/kernel_approximation.py#L829
it makes people think precomputed is supported. 
<!--
Tell us about the confusion introduced in the documentation.
-->

#### Suggest a potential alternative/fix
Change the _get_kernel_params method accordingly. 
Or alternatively add precomputed kernel support in the Nystroem class. But the fix seems to be requiring some change (since now in line 773 we should do basis_kernel = precomputed_kernel[basis_inds][:, basis_inds]), thus we need an extra parameter for passing the precomputed kernel. 
<!--
Tell us how we could improve the documentation in this regard.
-->

user: ZuowenWang0000
issue_19107: state: open
title: add plot_contingency_matrix / ContingencyMatrixDisplay
body: I think it would be great if we could add plotting capability for contingency matrices similar to the ones for confusion matrices.
The main difference is that confusion matrices assume that the two sets of classes are the same, while contingency  matrices are between arbitrary sets of categories.
This should be a relatively straight-forward but useful addition.
user: amueller
issue_19087: state: open
title: FIX Isolation forest path length for small samples
body: #### Reference Issues/PRs

Fixes #15724: iForest average path length for small samples (sklearn.ensemble.IsolationForest)
Also, to mitigate the numerical instability issue outlined in #16721 and #16967, a threshold value is implemented as suggested in the aforementioned PRs.

#### What does this implement/fix? 
An improved approximation for the average path length in isolation forests is introduced, based on more a accurate harmonic number calculation. The routine _average_path_length() is mostly replaced. This leads to more accurate anomaly scores.

#### Any other comments?
For the harmonic number calculation, see Wikipedia (https://en.wikipedia.org/wiki/Harmonic_number#Calculation) or the following publications and references therein. Further improvement regarding accuracy and/or performance may be feasible using e.g. a Ramanujan expansion.
Villarino, M. Ramanujan's Harmonic Number Expansion into Negative Powers of a Triangular Number. https://arxiv.org/abs/0707.3950
or
Wang, W. Harmonic Number Expansions of the Ramanujan Type. Results Math 73, 161 (2018). https://doi.org/10.1007/s00025-018-0920-8



user: Konrad0
issue_19081: state: open
title: KBinsDiscretizer min, max, median, mean, mode encoders
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
Being able to encode real values matrices using KBinsDiscretizer in a way to encode it with minimum, maximum, median, mean or mode value in training set.

#### Describe your proposed solution
Adding 5 new strategies to `sklearn.preprocessing.KBinsDiscretizer`.
Example given:
![image](https://user-images.githubusercontent.com/9883873/103347177-4efb0680-4aa7-11eb-9bee-ca919ec18505.png)
Code on image:
```
X = np.arange(30).reshape(-1, 2)
p = KBinsDiscretizer(n_bins=5, encode='min')
print(X)
p.fit_transform(X)
```
Should work as:
```
X = np.arange(30).reshape(-1, 2)
p = KBinsDiscretizer(n_bins=5, encode='ordinal')
print(X)

Xt = np.empty_like(X)
labels = p.fit_transform(X).astype(int)
for jj in range(X.shape[1]):
    column = X[:, jj]
    label = labels[:, jj]
    for value in range(labels.max() + 1):
        mask = label == value
        Xt[mask, jj] = min(column[mask])
Xt
```

#### Describe alternatives you've considered, if relevant
https://github.com/sasasagagaga/BinsEncoder

#### Additional context

user: CrafterKolyan
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
Needs Decision - Include Feature: Requires decision regarding including feature
issue_19079: state: open
title: [MRG] Correct handling of missing_values and NaN in SimpleImputer for object arrays (closes #19071)
body: #### Reference Issues/PRs

Fixes #19071.
In `SimpleImputer` strategies mean/median you can use input arrays with dtype is `object`. Imagine numeric data in the input array, except for missing values. In object arrays, you have a lot of flexibility on how to encode missing values. You can use `np.nan`, `None`, a string constant or anything else. There are two scenarios which are described in the issue #19071:

1.  There is more than one type of missing value in the input object array, one of them is specified as the `missing_values` parameter of the `SimpleImputer`. In this case the mean/median imputation should fail, because there are still non-numeric values in the input array other than `missing_values` which is masked. See #19071 for an example of not failing.
2. There is just one type of missing value in the input object array and that is specified as the `missing_values` parameter of the `SimpleImputer`. In this case the mean/median imputation must not fail and should calculate mean/median from the remaining values of the input array. See #19071 for an example of this failing with `missing_values=None`.

#### What does this implement/fix? Explain your changes.

In the `_validate_input` method, I keep the dtype as object.
In the `_dense_fit` method I try to convert the object array into float64 to prepare for mean/median calculation. If the not-masked values still contain some non-numeric values (scenario 1), ValueError is raised with the following informative error message:
```
Non-numeric values other than missing_values={missing_values}, showing {num_show}/{num_notmasked_nan}: {examples}
```
where the number of examples is limited to three.

#### Any other comments?

A comprehensive set of unit tests is added, which covers both scenarios.
user: ftrojan
issue_19076: state: open
title: Outside functionality:  add a "fitter" generator as an alternative to "fit" method
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable

At the current time, Estimators has a "fit" method. 

The fact that this is a method makes hard to "publish" or "monitor" the progress and evoluton of the internal state of the Estimator. 

Now, having a "fitter" method as a generator (with _yield_ statement) will leverage huge advantages over having to use specialized and standardized callbacks having to infer the effect of this callback in the internal loops and with little control on the access of the Estimator internal state.

#### Describe your proposed solution
Define a protocol to have the option to include "fitter()" generator as well, so people can perform checks every iteration (or every n%x==0) iterations and then have the option to plot, monitor, and use the internal state and params of the Estimator.

```
for estimator_output in estimator.fitter(data):
    print(estimator_output)
    pyplot.plot(estimator_output.parameters[0])
    estimator.learning_rate/=0.9
    if estimator_output.error>12:
        break
```

- This will provide a rich way of doing experiments and don't spend much time in creating sophisticated callback.
- This will provide the possibility to stop iterating when user wants to without "ctrl+c"
- This will help alleviate some black box feeling about "fit" methods

#### Describe alternatives you've considered, if relevant
I don't have come with a way of doing this without the "fitter" generator

#### Additional context

- This will help even more debugging of current algorithm implementations
- This can even be used to allow the interaction of two fitters comparing its behavior

#### Finally
In resume I consider it will change the way people see, understand and interact with machine learning algorithms implemented in the sklearn interface.

user: jamartinh
issue_19072: state: open
title: GroupedTimeSeriesSplit with params - gap and test_size
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

####
To include `gap` and `test_size` on the  `GroupTimeSeriesSplit` function  ( PR #16236 ) to stay consistent with the params in class  `TimeSeriesSplit`. 
This Feature request is in response to a  [comment ](https://github.com/scikit-learn/scikit-learn/pull/16236#issuecomment-703249182) by @albertvillanova  on PR #16236   for feature #14257.  
The implementation was left for a future PR **pending discussion** on the desired behavior . 

- Param `gap` shall add  a gap between the train group and the test group while preserving the temporal order and group boundaries. 
- Param `test_size` shall limit the size of the test_group with each split. 

#### On a recent [Kaggle Competition](https://www.kaggle.com/c/jane-street-market-prediction)  this modification  was implemented    by  user : [marketneutral](https://www.kaggle.com/marketneutral)   in [this Notebook ](https://www.kaggle.com/marketneutral/purged-time-series-cv-xgboost-optuna) . The name of the class on the notebook  is `PurgedGroupTimeSeriesSplit`

If the above notebook captures  the idea as intended then it can be included in a PR as is . 


cc: @jnothman  , @ogrisel 

user: getgaurav2
issue_19071: state: open
title: SimpleImputer, missing_values and None
body: There are issues with `SimpleImputer` and None:

1. None values are treated as NaNs even when the dtype is `object`. Unlike the `float` dtype, `object` can support both nans **and** None so there's no reason to treat them the same.

```py
import numpy as np
from sklearn.impute import SimpleImputer

a = np.arange(5).reshape(-1, 1).astype(object)
a[3] = None

# This will not fail and impute None as if it were nan
SimpleImputer(missing_values=np.nan).fit_transform(a)
```

2. Unlike what the doc says, `missing_values=None` is not supported. In fact, `missing_values` is expected to be of numerical dtype, not `NoneType`. The error message isn't super descriptive:

```py
import numpy as np
from sklearn.impute import SimpleImputer

a = np.arange(5).reshape(-1, 1).astype(object)
a[3] = None

SimpleImputer(missing_values=None).fit_transform(a)
```
fails with:

`ValueError: Input contains NaN, infinity or a value too large for dtype('float64').`

(I haven't checked other imputers which may or may not have similar issues)
user: NicolasHug
issue_19066: state: open
title: BallTree query match time is O(n) not O(log(n))
body: I've run performance analysis on matching NN with BallTree (same with KDTree), and the matching time is linear to number of elements, and should be O(log(n)). 

Here are the result of benchmark:
num_elements, match_time
10000 0.09097146987915039
20000 0.18194293975830078
40000 0.3668830394744873
80000 0.7527577877044678

Here is my code:
```

from sklearn.neighbors import BallTree
import numpy as np 
import time 


def tree_perf(tree_size):
    X = np.random.rand(tree_size, 512)
    Y1 = np.random.rand(1, 512)
    Y2 = np.random.rand(10, 512)

    ts = time.time()
    kdt = BallTree(X, leaf_size=30, metric='euclidean')
    load_tree = time.time() - ts
    num_nn = 1
    ts = time.time()
    vs = kdt.query(Y1, k=num_nn, return_distance=True)
    match1 = time.time() - ts
    ts = time.time()
    vs = kdt.query(Y2, k=num_nn, return_distance=True)
    match10 = time.time() - ts
    print(tree_size, load_tree, match1, match10)

print("num_elements", "load_tree", "match_1", "match_10")

for i in range(100):
    tree_size = 10000 + i * 10000
    tree_perf(tree_size)
```
user: simsim314
issue_19065: state: open
title: Add resample to AdaBoostClassifier in _weight_boosting.py
body: My idea: 
Add the feature of resample (like in Bagging Classifier) to AdaBoost. In the literature this is quite common.
In every iteration of the n_estimators Boosting n_samples will be randomly picked and used for the training of the base estimator but the error score evaluation will be performed on the whole data set. 
This way overfitting is largely reduced. I use this modification myself.

user: phbk
issue_19045: state: open
title: Better (__-free) ways to specify grid search hyperparameters
body: #### Describe the workflow you want to enable

I developed [searchgrid](http://github.com/jnothman/searchgrid) to deal with one of my pet peeves in scikit-learn: that when performing a hyperparam search you need to specify complicated `__`-delimited paths to parameters that reflect a composite estimator structure defined elsewhere.

I think searchgrid does the right thing by allowing you to specify the parameter grids _locally_ with respect to the estimator in which each parameter is changed.

This benefits in a few ways:
* when you change (e.g. add a layer of nesting to) your pipeline, you don't need to change your parameter grid specification. I find this really annoying.
* it's easy to specify parameter grids where some component of a pipeline has multiple alternative estimators for a step.

Problems with searchgrid include:
* It stores parameter grid settings as an attribute each estimator, which may be confusing since these settings only have any effect when a parameter grid is constructed from the estimators.
* It only currently supports GridSearchCV (although HalvingGridSearchCV would be an easy extension), though it could be extended to some distribution-based searches.
* I've not maintained it much (and it includes redefinition of make_pipeline, etc, which have been changed in scikit-learn), and it's not widely known about.

I regularly – at least when I work with someone else's code – feel the `__`-based parameter specification is a usability frustration always there when building and modifying composite estimators.

#### Describe your proposed solution

Either:

1. Adopt something very similar to searchgrid's interface, allowing a parameter grid or distribution to be specified on each constituent estimator. param_grid is then not needed when constructing GridSearchCV, since the grid is found on the base estimator.
      ```py
	  kbest = set_grid(SelectKBest(), k=[5, 10, 20])
	  pca = set_grid(PCA(), n_components=[5, 10, 20])
	  lr = set_grid(LogisticRegression(), C=[.1, 1, 10])
	  pipe = set_grid(Pipeline([('reduce', None),
	                            ('clf', lr)]),
	                  reduce=[kbest, pca])
	  gs = make_grid_search(pipe)
      ```
2. Allow an alternative specification of parameter grid / distribution, as a mapping of `{(est, param_name): [value1, value2, ...], ...}`, which is equivalent to what's used in searchgrid, but doesn't set any attributes on the estimators themselves, and puts the grid specification into the grid search rather than when constructing the composite estimator.
      ```py
	  kbest = SelectKBest()
	  pca = PCA()
	  lr = LogisticRegression()
	  pipe = Pipeline([('reduce', None),
                       ('clf', lr)])
	  gs = GridSearchCV(pipe, param_spaces={
          (pipe, 'reduce'): [kbest, pca],
          (kbest, 'k'): [5, 10, 20],
          (pca, 'n_components'): [5, 10, 20],
          (lr, 'C'): [.1, 1, 10],
      })
      ```

#### Additional context

* Searchgrid also gets rid of the need for step names in pipelines with a syntax like `make_pipeline([kbest, pca], lr)`.
* If the user no longer sees `__` notation in specification, it would also be good to solve https://github.com/jnothman/searchgrid/issues/5, which says that we also need to get rid of `__` notation in `cv_results_`. One option might just be for `*SearchCV` to provide a mapping between `(estimator, param_name)` tuples and `__` notation for the same parameter.
user: jnothman
issue_19043: state: open
title: added ValueError for improperly shaped Y when doing a multilabel fit
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
See also #18947 for related conversation. These changes were initially on that PR
but should have been on their own.


#### What does this implement/fix? Explain your changes.
Adds a raise ValueError check for multi-label fits where Y is not truly multi-labeled.
Passing a 1-dimensional Y fails since it isn't multi-labeled, so this change enforces
the correct dimensions of Y and adds a more helpful error message.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: efiegel
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_19042: state: open
title: CCA axes depend on the order of columns in Y matrix
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
<!--
A clear and concise description of what the bug is.
-->
Changing the order of response variables in the Y matrix results in different selection of CCA axes. The loading vector of the first response variable is always parallel to the first CCA axis.
#### Steps/Code to Reproduce
<!--
Please add a minimal example that we can reproduce the error by running the
code. Be as succinct as possible, do not depend on external data. In short, we
are going to copy-paste your code and we expect to get the same
result as you.

Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
docs = ["Help I have a bug" for i in range(1000)]
vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)
lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

```python
import numpy as np
from sklearn.cross_decomposition import CCA

# Generate random data
np.random.seed(0)
data = np.random.rand(10, 10)

# Generate random set of response variables
np.random.seed(1)
test = np.random.rand(10, 6) 

# Change order of response variables
test2 = np.copy(test)
test2[:, 3:] = test[:, :3]
test2[:, :3] = test[:, 3:]

# Perform the CCA for both response variable datasets
cca = CCA(n_components=2)
cca.fit(data, test)
d = cca.transform(data)
y = cca.y_loadings_

cca2 = CCA(n_components=2)
cca2.fit(data, test2)
d2 = cca2.transform(data)
y2 = cca2.y_loadings_

# Print first five rows of X scores and first Y loading vector for both cases
print(d[:5])
print(d2[:5])
print(y[0])
print(y2[0])
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
As both **cca** and **cca2** objects are built on the same data, I expect the resulting CCA scores of observations (**d** and **d2**) be identical or nearly so in both cases. The direction of the loading vector of the first response variable (**y** and **y2**) is most probably not parallel to any of the "real" CCA axes, so I do not expect the loadings close to 0, 1 or -1.
#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
The scores in the first rows of **d** and **d2** are obviously different. In my case the first five rows are like

```
[[-0.13226527  0.39666035]
 [ 0.47686754 -0.44630239]
 [ 0.42884673 -0.56030583]
 [ 0.59795741 -0.61033328]
 [-1.34483863 -0.74278071]]
```
and
```
[[-0.22933487  0.76123766]
 [ 0.03491544  0.23805463]
 [ 0.18202364  0.35970863]
 [ 0.51478239  0.20467545]
 [-0.52352471 -0.12359243]]
```
, respectively.

Both **y** and **y2** loading vectors for the first variable are parallel to the first CCA axis or nearly so:
`[-1.000000e+00 -4.188481e-15]`
and
`[ 1.00000000e+00 -1.00802462e-14]`
, respectively.
#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
import imblearn; print("Imbalanced-Learn", imblearn.__version__)
-->
System:
    python: 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]
executable: c:\anaconda3\python.exe
   machine: Windows-10-10.0.18362-SP0

Python dependencies:
          pip: 20.3
   setuptools: 50.3.2.post20201201
      sklearn: 0.23.2
        numpy: 1.19.2
        scipy: 1.5.2
       Cython: 0.29.21
       pandas: 1.1.3
   matplotlib: 3.1.3
       joblib: 0.17.0
threadpoolctl: 2.1.0

Built with OpenMP: True

<!-- Thanks for contributing! -->

user: taprs
issue_19030: state: open
title: Mixture Entropy Estimation for Gaussian Mixture Models
body: #### Describe the workflow you want to enable
I would like to estimate the differential entropy of a Gaussian Mixture Model (GMM). Knowing the entropy of a model can help in model selection as generally models with higher entropy best suit the data ([Principle_of_maximum_entropy](https://en.wikipedia.org/wiki/Principle_of_maximum_entropy)). Unfortunately there is no analytical solution to compute differential entropy for GMMs.

#### Describe your proposed solution
I implemented the entropy estimators for GMM found in the paper ["Estimating Mixture Entropy with Pairwise Distances"](https://arxiv.org/abs/1706.02419)[1]: one estimates an entropy lower bound and one estimates an upper bound. The estimators use pairwise distances between multivariate Gaussians, one uses Chernoff-divergence one uses Kullback-Leibler-divergence (these have closed-form expressions). The method does not require any additional variable, it simply works by using the weights_, means_, covariances_, precision_ existing attributes of the GMM. I am not an expert but it seems from the paper [1] that these bounds, other than entropy estimation, might also provide additional insights on how well the GMM suits the data based on the spread between lower and upper bound (tight bound->better fit).

#### Describe alternatives you've considered, if relevant
One could estimate the entropy by computing the log-likelihood of a **Large** set of points sampled from the GMM (Monte Carlo estimate). For a large number of samples the estimate should approximate the actual entropy. This becomes relatively slow for growing number of components and in particular number of features.

#### Additional context
I have implemented this already and I'd be happy to open a PR but I wanted to hear whether this is something that it is worth to integrate in scikit-learn. The paper mentioned above has ~60 citations on scholar so far and is from 2017. I briefly tested the execution times and the method mentioned above becomes quite faster and more accurate than Monte Carlo for growing number of features, I can expand on these tests to check the performance improvements.

[1] - Kolchinsky, Artemy, and Brendan D. Tracey. "Estimating mixture entropy with pairwise distances." Entropy 19.7 (2017): 361.


user: giuliolovisotto
issue_19015: state: open
title: Include MStream in Sklearn
body: Hi,

According to the scikit-learn inclusion criterion, "A technique that provides a clear-cut improvement (e.g. an enhanced data structure or a more efficient approximation technique) on a widely-used method will also be considered for inclusion."
I am submitting this request to include MStream:

Code: https://github.com/Stream-AD/MStream/
Research Paper: https://arxiv.org/pdf/2009.08451.pdf

Thank you!

user: bhatiasiddharth
issue_19012: state: open
title: Confusion Matrix Representation / Return Value
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable

An enhancement to the _**output**_ of confusion matrix function, better representing the true and predicted values for multilevel classes.
- i.e. Current Representation with code:
`from sklearn.metrics import confusion_matrix`
`y_true = ["cat", "ant", "cat", "cat", "ant", "bird"]`
`y_pred = ["ant", "ant", "cat", "cat", "ant", "cat"]`
`confusion_matrix(y_true, y_pred, labels=["ant", "bird", "cat"])`
_Output:_
**array([[2, 0, 0],
       [0, 0, 1],
       [1, 0, 2]])**


#### Describe your proposed solution
When you have multiple levels you can have difficulty reading the ndarray, associating the levels with the True and Predicted values.

Proposed solution should look similar to the table below, providing better readability of the confusion matrix.
&nbsp;|&nbsp;|**_Predicted_**|**_Value_**|&nbsp;
|-| -| - | - | - |
|&nbsp;|**_Levels_**|ant|bird|cat
|**_True_**|ant|2| 0| 0
|**_Value_**|bird|0| 0| 1
|&nbsp;|cat|1| 0| 2

Possible Solutions:
1. Provide a parameter to prettyprint the matrix. printMatrix [type:bool]
2. Include another parameter to return ndarray, index as _true_values_, columns as _predicted_values_
For example:
`cm = confusion_matrix(y_true, y_pred, labels=["ant", "bird", "cat"])`
`index=["true:ant", "true:bird", "true:cat"]`
`columns=["pred:ant", "pred:bird", "pred:cat"]`
`return cm, index, columns`
Which can be easily converted into a dataframe for further use
 


user: shubhamdo
issue_19008: state: open
title: MIN_CAT_SUPPORT in HGBT
body: #### Background
One leftover of categorical support in `HistGradientBoostingRegressor` and `HistGradientBoostingClassifier`, merged in #18394, is the `MIN_CAT_SUPPORT` in splitting.pyx, see https://github.com/scikit-learn/scikit-learn/pull/18394#discussion_r523794802. It acts as a smoothing parameter to categorical variables and is currently fixed to the value `10`.

#### Proposal
I propose to investigate the impact of `MIN_CAT_SUPPORT`. In case of a larger impact it would be desirable to pass this as an option to the user.

#### Additional context
AFAIK, very little reproducible results are known to date. For references, see the comment linked above.
user: lorentzenchr
labels: Moderate: Anything that requires some knowledge of conventions and best practices
Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_18997: state: open
title: META Quantile Regression
body: This meta issue aims at giving an overview/summary of all quantile related issues and pull requests. This may help to coordinate future work.

- [x] `metrics`: Add quantile loss as metric #18911, done in #19415
- [x] `linear_model`: Add quantile regression ~~#3148~~, also ~~#13612~~
      PR: #9978 and ~~#16343~~
- [ ] `tree` and `ensemble` (based on `DecisionTreeRegressor`): #4210, #4768, #11086 #18540
- [x] `HistGradientBoostingRegressor`: #17955

Also related is naming of losses #18248, and a possible private loss function module #15123.
user: lorentzenchr
issue_18970: state: open
title: Implement binary/multiclass classification metric - Spherical Payoff
body: #### What does this implement? Explain your changes.
This PR implements spherical payoff score for binary/multiclass classification.

This metric is most used in ecological modelling for Decision Making models such as Bayesian Networks, where you want to maximize the correct predicted classes and the model's confidence in that output.

 It's measures the model's confidence, through the probability, to predict the correct category and have a defined interval: [0, 1]. Best possible score is 1.0, and the worst is 0.0. It's calculated as the average over all samples.

The benefits to use this score:
- See if your model have confidence in predicted answers
- Support other useful metrics such as Confusion Matrix related, given the fact that measures the output probabilities
- The other metrics equivalents (scoring rules metrics, read more [here](https://en.wikipedia.org/wiki/Scoring_rule#:~:text=In%20decision%20theory%2C%20a%20score,set%20of%20mutually%20exclusive%20outcomes.)) as brier score or log loss it's only available for binary problems.  This metric doesn't have this issue.

#### Any other comments?
The equation was implemented based on this [2007 paper reference](https://www.fs.fed.us/pnw/pubs/journals/pnw_2006_marcot001.pdf):
![equation](https://user-images.githubusercontent.com/32513366/101266922-2dba2b00-3732-11eb-908a-d8a293a2780b.png)


Other point is the fact that **was not implemented yet for open source community**, the only reference that I found was in a paid program call [Netica in this tutorial](https://www.norsys.com/tutorials/netica/secD/tut_D2.htm).

Abs,

user: KaikeWesleyReis
issue_18966: state: open
title: t-SNE runs out of memory just before returning results and after finishing computation
body: #### Describe the bug

Based on the verbose output of TSNE it seems that the whole computation actually finishes fine and just when it is about to return the results it writes a large amount of data to RAM and then gets killed by the system.

#### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.manifold import TSNE

large_N = 100_000  # for my system this is large enough, of course smaller N like 10_000 work
pca_result = np.random.normal(size=(large_N, 50))
tsne = TSNE(n_components=2, verbose=10, perplexity=40, n_iter=300)
tsne_pca_results = tsne.fit_transform(pca_result)
```

#### Expected Results
Running this should behave the same way as for smaller N like 10.000, or, run out of memory during execution, because the TSNE needs more memory to run for given N than provided by the system.

#### Actual Results
Following the verbose output and the runtime, even for the given large_N of 100.000 TSNE (which I believe uses BarnesHut implementation as default) is actually able to index the data and compute conditional probabilities. The verbose output also shows the running of 300 iterations with reduced error each time. The last line of output shows the achieved the KL divergence. At this point I believe TSNE has actually run successfully and should just return embedding matrix of shape (N,2) which is much smaller than (N,50) and should therefore have no problem to fit into memory. But when opening `htop` in a parallel terminal one can actually observe how the program writes data into the RAM within seconds, fills it completly up and then gets killed by the system. Below the output as observed on my terminal:

```
t-SNE] Computing 121 nearest neighbors...                                                                                                                                                  | 0/202 [00:00<?, ?it/s]
[t-SNE] Indexed 535327 samples in 5.554s...                                                                                                                                                                        
[t-SNE] Computed neighbors for 535327 samples in 57.435s...                                                                                                                                                        
[t-SNE] Computed conditional probabilities for sample 1000 / 535327                                                                                                                                                
[t-SNE] Computed conditional probabilities for sample 2000 / 535327                                                                                                                                                
[t-SNE] Computed conditional probabilities for sample 3000 / 535327                                                                                                                                                
[t-SNE] Computed conditional probabilities for sample 4000 / 535327                                                                                                                                                
[t-SNE] Computed conditional probabilities for sample 5000 / 535327                                                                                                                                                
[t-SNE] Computed conditional probabilities for sample 6000 / 535327                                                                                                                                                
[t-SNE] Computed conditional probabilities for sample 7000 / 535327                                                                                                                                                
[t-SNE] Computed conditional probabilities for sample 8000 / 535327
...
[t-SNE] Computed conditional probabilities for sample 534000 / 535327
[t-SNE] Computed conditional probabilities for sample 535000 / 535327
[t-SNE] Computed conditional probabilities for sample 535327 / 535327
[t-SNE] Mean sigma: 0.000000
[t-SNE] Computed conditional probabilities in 70.161s
[t-SNE] Iteration 50: error = 123.6066971, gradient norm = 0.0000000 (50 iterations in 89.115s)
[t-SNE] Iteration 50: gradient norm 0.000000. Finished.
[t-SNE] KL divergence after 50 iterations with early exaggeration: 123.606697
[t-SNE] Iteration 100: error = 7.7342658, gradient norm = 0.0000001 (50 iterations in 101.858s)
[t-SNE] Iteration 150: error = 7.7342658, gradient norm = 0.0000067 (50 iterations in 110.148s)
[t-SNE] Iteration 200: error = 7.7170248, gradient norm = 0.0012155 (50 iterations in 119.877s)
[t-SNE] Iteration 250: error = 6.7275314, gradient norm = 0.0006462 (50 iterations in 91.746s)
[t-SNE] Iteration 300: error = 4.7438197, gradient norm = 0.0004072 (50 iterations in 57.282s)
[t-SNE] KL divergence after 300 iterations: 4.743820
Killed
```

#### Versions

Output of `import sklearn; sklearn.show_versions()`:
```
System:
    python: 3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]
executable: /lhome/davidj2/code/sync/pointset_attention_net/.venv/bin/python3.6
   machine: Linux-4.15.0-108-generic-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 20.2.4
   setuptools: 50.3.2
      sklearn: 0.23.2
        numpy: 1.19.4
        scipy: 1.5.3
       Cython: 0.29.21
       pandas: 1.1.4
   matplotlib: 3.3.3
       joblib: 0.17.0
threadpoolctl: 2.1.0

Built with OpenMP: True
```
user: demmerichs
issue_18953: state: open
title: Improve dense SVC performance by reimplementing dense SVC rbf kernel with GEMV BLAS API
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

#### What does this implement/fix? Explain your changes.
Re-implement SVC rbf train/predict kernel with GEMV BLAS API instead of currently used DOT. 

This new implementation can greatly improve performance for both training and predicting. We picked 120+ datasets from https://archive.ics.uci.edu/ml/datasets and kaggle.com, part of the datasets are: abalone, statlog serials, musk serials, breast-cancer serials, etc. As an overall, this new implementation provides ~1.7x (training)/~2.5x (predict) performance than the master code on a AVX512 supported server. Generally, the bigger the dataset (more samples) the bigger benefit will be. The benefit mainly comes from GEMV vs. DOT ---- GEMV as level 2 BLAS API is much more efficient than DOT which is level 1 BLAS API when the input data is not too small. 

#### Any other comments?
Shrinking needs be disabled during runtime to make this new kernel workable, or it will fall back to current dot based kernels. User can specify shrinking to be 0 when new a SVC instance.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Guobing-Chen
issue_18943: state: open
title: plot_tree showing values based on class_weights
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
<!--
A clear and concise description of what the bug is.
-->
The `values` in the plot_tree are not represent the actual count, whereas represent the values generated based on class_weights. Is this the expected behaviour?

Actually for me, it felt confusing because the sum of the`values` is greater than the `samples`

#### Steps/Code to Reproduce


```
import matplotlib.pyplot as plt

from sklearn.datasets import make_classification
from sklearn.tree import DecisionTreeClassifier, plot_tree

X, y = make_classification(random_state=42)
dt = DecisionTreeClassifier(class_weight={0:1, 1:2},random_state=42).fit(X,y)
plot_tree(dt, fontsize=5, filled=True)
plt.show()
```
<img width="253" alt="image" src="https://user-images.githubusercontent.com/20812407/100594851-3371d400-3320-11eb-8f9e-9a90b14ad573.png">

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
Not modifying the `values` based on the `class_weights` and it's sum should be equal to samples.


#### Versions
System:
    python: 3.8.3 (default, May 19 2020, 13:54:14)  [Clang 10.0.0 ]
executable: /opt/anaconda3/envs/so/bin/python
   machine: macOS-10.14.6-x86_64-i386-64bit

Python dependencies:
          pip: 20.0.2
   setuptools: 46.4.0.post20200518
      sklearn: 0.23.1
        numpy: 1.18.1
        scipy: 1.4.1
       Cython: None
       pandas: 1.0.3
   matplotlib: 3.1.3
       joblib: 0.15.1
threadpoolctl: 2.1.0

Built with OpenMP: True
<!-- Thanks for contributing! -->

user: venkyyuvy
issue_18942: state: open
title: FIX Improved the error message from check_classification_targets function when label is invalid
body: When invalid types of labels are used, the error `Unknown label type` is raised.
 This pull request explicitly added the expected label types to the error message, which should provide more actionable information (especially for novices).

**Expected error message:**
`"Unknown label type: %r. Expected label types include 'binary', 'multiclass', 'multiclass-multioutput', 'multilabel-indicator', 'multilabel-sequences' % y_type"`

`"Unknown label type: %s. Expected 'binary', 'multiclass', 'multilabel'" % repr(ys)`


user: ZhihuiChen0903
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_18941: state: open
title: .fit.transform != .fit_transform inconsistency in PCA results
body: PCA's fit_transform returns different results than the application of fit and transform methods individually. A piece of code that shows the inconsistency is given below.

```python
import numpy as np
from sklearn.decomposition import PCA

nn = np.array([[0,1,2],[3,4,5],[6,7,8]])
pca = PCA(n_components=2, random_state=42)
print(pca.fit_transform(nn))

nn = np.array([[0,1,2],[3,4,5],[6,7,8]])
pca = PCA(n_components=2, random_state=42)
pca.fit(nn)
print(pca.transform(nn))
```

Please run the code with sklearn 0.23.2


user: tdincer
issue_18936: state: open
title: FEA SLEP006: debugging tool for sample props
body: As stated by @jnothman in https://github.com/scikit-learn/scikit-learn/pull/16079#issuecomment-735210136, we should have a debugging tool to show users how metadata is routed around.

Having this as a separate issue since it requires a rather detailed discussion of its own.
user: adrinjalali
issue_18934: state: open
title: AdaBoostClassifier / BaseWeightBoosting no longer supports negative sample weights
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
<!--
A clear and concise description of what the bug is.
-->

The commit 1888a96908301a9d71e37ce8c08bfd64298e9192 (change made to `sklearn/ensemble/_weight_boosting.py`, corresp. PR #14294) broke support for negative sample weights in AdaBoost.  Previously, these were supported as long as their sum was not negative. Negative sample weights are commonly produced by statistical methods such as sPlot, or Monte Carlo generators in HEP.

My motivation for raising this issue is well discussed in issue #3774.

Perhaps the only change that needs to be made is:

this block in `sklearn/ensemble/_weight_boosting.py`
```python
        if np.any(sample_weight < 0):
            raise ValueError("sample_weight cannot contain negative weights")
```
is changed back to
```python
        if sample_weight.sum() <= 0:
            raise ValueError(
                    "Attempting to fit with a non-positive "
                    "weighted number of samples.")
```

Many thanks in advance!

#### Steps/Code to Reproduce

Any use of `AdaBoostClassifier` where `sample_weight` passed to `fit(...)` has negative sample weights. (i.e. `np.any(sample_weight < 0)` is `True`)

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->

No error is thrown

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
```
Traceback (most recent call last):
... snip ...
  File "workflow/scripts/xicp_mva.py", line 320, in train_model
    clf = model.fit(X_train, y_train, sample_weight=w_train)
  File "/home/roneil/excitedxic-plus/.snakemake/conda/8151d681/lib/python3.8/site-packages/sklearn/ensemble/_weight_boosting.py", line 443, in fit
    return super().fit(X, y, sample_weight)
  File "/home/roneil/excitedxic-plus/.snakemake/conda/8151d681/lib/python3.8/site-packages/sklearn/ensemble/_weight_boosting.py", line 114, in fit
    raise ValueError("sample_weight cannot contain negative weights")
ValueError: sample_weight cannot contain negative weights
```
#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
import imblearn; print("Imbalanced-Learn", imblearn.__version__)
-->

```
    python: 3.8.6 | packaged by conda-forge | (default, Oct  7 2020, 19:08:05)  [GCC 7.5.0]
executable: /home/roneil/excitedxic-plus/.snakemake/conda/8151d681/bin/python
   machine: Linux-5.9.10-200.fc33.x86_64-x86_64-with-glibc2.10

Python dependencies:
          pip: 20.2.4
   setuptools: 50.3.1.post20201107
      sklearn: 0.23.2
        numpy: 1.19.4
        scipy: 1.5.3
       Cython: 0.29.21
       pandas: 1.1.4
   matplotlib: 3.3.3
       joblib: 0.17.0
threadpoolctl: 2.1.0

Built with OpenMP: True
[1087168323 1165323093 2474382879 ...  431983649  423404152  447327467]
[     0      1      2 ... 392744 392748 392750]
```

<!-- Thanks for contributing! -->

user: ryuwd
labels: Needs Decision: Requires decision
issue_18933: state: open
title: MDS implementation has error in smacof algorithm for non-metric case
body: Dear Scikit team,
I really enjoy using the scikit learn library and I am very greatful for your very impactful work. I hope I can contribute with this bug report (if it is true) to this library.
Keep on your good work!

I checked existing issues that concern MDS it is not the same as #16846 but to solve it I would also address #11381

#### Describe the bug
Everything concerns the MDS implementation in `scikit-learn/sklearn/manifold/_mds.py`. I have copied the relevant snipped of code below and added the line numbers I am referring to. All equations I am referring to are from [1]

__Short version:__
- The initialization of the disparities in [line 103](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/manifold/_mds.py#L103) leads to an error when calculating the stress and the update of the configuration.
- The standardization in [line 106](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/manifold/_mds.py#L106) is wrong.
- The library can not reproduce results from the [r library mass](https://stat.ethz.ch/R-manual/R-devel/library/MASS/html/isoMDS.html)

__Long version:__
I found the following error in the implementation:
- In `scikit-learn.sklearn.manifold._mds._smacof_single` in [line 103](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/manifold/_mds.py#L103) the disparities get initialized with the distances. In [line 104](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/manifold/_mds.py#L104) only the values of the upper triangle matrix get overwritten by the calculated disparities (`disparities_flat`). This results in a matrix that has disparities (similar to rank distances) in the upper triangle and euclidean distances in the lower triangle. Therefore the normalization factor in [line 106](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/manifold/_mds.py#L106) is calculated based on a matrix that is half distances and half disparities
- Line 106 standardized the disparities to 
![grafik](https://user-images.githubusercontent.com/31069329/100464473-f6160800-30cd-11eb-9118-94e754c2cc24.png)
which corresponds to step 3 and 9 of the `Smacof algorithm with Admissibly Transformed Proximities`[2]. However, from equation 9.1 it seems like this corresponds only to the upper triangle matrix (which explains the `n(n-1)/2`) in the implementation of line [106](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/manifold/_mds.py#L106), all elements from the matrix are used which results in a wrong standardization factor.
- Also [line 106](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/manifold/_mds.py#L106) the term ```(disparities ** 2).sum()``` should not be in the square-root. This results from a transformation when we are looking for a normalization factor `a` that is applied to the disparities such that:  
![grafik](https://user-images.githubusercontent.com/31069329/100475051-64fd5c00-30e2-11eb-87b5-45bdeb3e723f.png)
We then should calculate `a` as 
![grafik](https://user-images.githubusercontent.com/31069329/100475075-7b0b1c80-30e2-11eb-9e4a-b9e8b97b3f50.png)
(equations 9.1 + 9.2) from [1]

- When the stress is computed in [line 110](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/manifold/_mds.py#L110), a difference between the dissimilarities and the disparities is computed. As stated above, the disparities had the dissimilarities in the lower triangle matrix (which I think is confusing) which would normally mean that the all differences in the lower triangle matrix are zero. However, as the disparities got scaled in [line 106](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/manifold/_mds.py#L106), the differences in the lower triangle matrix are no longer zero but some (more or less) random values. 


There are also some smaller issues that I noticed:
- After the last update of the configuration, the stress is not updated. This means the old stress is returned (I think this is the same as #16846)
- I think it would be good to return the normalized stress 

__References__
[1]:  “Modern Multidimensional Scaling - Theory and Applications” Borg, I.; Groenen P. Springer Series in Statistics (1997), Version from 2005
[2]: “Modern Multidimensional Scaling - Theory and Applications” Borg, I.; Groenen P. Springer Series in Statistics (1997), Version from 2005, Page 204 in Chapter _9. Metric and non-metric MDS_

Code from [scikit MDS implementation](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/manifold/_mds.py#L89) (for reference)
```python
ir = IsotonicRegression()
    for it in range(max_iter):
        # Compute distance and monotonic regression
        dis = euclidean_distances(X)

        if metric:
            disparities = dissimilarities
        else:
            dis_flat = dis.ravel()
            # dissimilarities with 0 are considered as missing values
            dis_flat_w = dis_flat[sim_flat != 0]

            # Compute the disparities using a monotonic regression
            disparities_flat = ir.fit_transform(sim_flat_w, dis_flat_w)
[103]    disparities = dis_flat.copy()
[104]    disparities[sim_flat != 0] = disparities_flat
[105]    disparities = disparities.reshape((n_samples, n_samples))
[106]    disparities *= np.sqrt((n_samples * (n_samples - 1) / 2) /
                                   (disparities ** 2).sum())

        # Compute stress
[110]    stress = ((dis.ravel() - disparities.ravel()) ** 2).sum() / 2

```

#### Steps/Code to Reproduce
The code and data to calculate the results is in this gist: (https://gist.github.com/henrymartin1/05a2aa5210e52216abd7ce61e6918e3a). The relevant file is`scikit_mds_bugreport.py`

I honestly had a hard time to come up with code that proofs that there is a problem. This is what I did: 
- I created an array and its distance matrix in python and stored it as .csv
- Using this data, I calculated the optimal configuration and the stress using the non-metric MDS implementation of the [r-library mass](https://www.rdocumentation.org/packages/MASS/versions/7.3-53/topics/isoMDS)
- I calculate the optimal configuration and the stress using scikit-learns mds
- I calculate the non-metric stress using my own implementation of non-metric stress
- I plot the optimal scikit configuration and the optimal r configuration

#### Expected Results
- The two optimal configurations would be the same or very similar
- The stress reported by r and the stress reported by scikit are the same
- The configurations found by scikit and by r have the same stress when I calculate it using my own implementation of the stress function. (+ they match with what is reported by scikit and the r implementation)

#### Actual Results
- The optimal scikit configuration and the optimal r configuration are significantly different (see plots below)
- My calculated stress for the r configuration and the scikit configuration are different
- The scikit stress is not normalized (not necessarily a problem)
##### Result mds scikit 
![grafik](https://user-images.githubusercontent.com/31069329/100474874-f8825d00-30e1-11eb-920d-c568ffe6e93c.png)
##### Result mds r
![grafik](https://user-images.githubusercontent.com/31069329/100474890-020bc500-30e2-11eb-8914-3ae4e61a486e.png)


#### Versions
System:
    python: 3.8.6 | packaged by conda-forge | (default, Oct  7 2020, 18:22:52) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\henry\.conda\envs\evhomepv\python.exe
   machine: Windows-10-10.0.19041-SP0

Python dependencies:
          pip: 20.2.4
   setuptools: 49.6.0.post20201009
      sklearn: 0.23.2
        numpy: 1.19.4
        scipy: 1.5.3
       Cython: None
       pandas: 1.1.4
   matplotlib: 3.3.2
       joblib: 0.17.0
threadpoolctl: 2.1.0

Built with OpenMP: True

<!-- Thanks for contributing! -->

user: henrymartin1
issue_18931: state: open
title: Make it possible to use ARPACK or randomized_svd solvers in PLSSVD
body: As initially discussed in https://github.com/scikit-learn/scikit-learn/pull/18302#discussion_r531530622 it might be interesting to add an extra constructor param to PLSSVD to select ARPACK or the sklearn's randomized_svd solver instead of the default LAPACK solver (from `scipy.linalg.svd`).

But the ARPACK and randomized_svd are non-deterministic so we would also need to add a `random_state` parameter.

Careful benchmarking to evaluate the speed vs numerical or statistical accuracy trade-off should be conducted to:

- help the user choose the value of this parameter (both in the docstring and the user guide)
- suggest an "auto" strategy to automatically select a good solver based on the shape of the data and the `n_components` parameter, similar to what is done in the `PCA` and `TruncateSVD` estimators. This "auto" parameter shall become the default after the usual deprecation period.
user: ogrisel
issue_18926: state: open
title: classification_report should support a beta argument, reporting f_beta instead of limited to f_1
body: #### Describe the workflow you want to enable
When iterating on models, we are often using the classification report to evaluate the strength of new features, building simple tabular models. 

I would like to be able to change the relative importance of recall to precision using the f_beta score instead of f_1.

#### Describe your proposed solution
I would like that `sklearn.metrics.classification_report` took a `beta` argument, to calculate the f_beta score rather than f_1. The function `precision_recall_fscore_support` that is used inside it, does take beta as an argument, so it would be easy to pass on the parameter to that function (and rename f1 to f_beta if beta != 1.)

#### Describe alternatives you've considered, if relevant
We could also use the `precision_recall_fscore_support` and create our own report downstream, but I think others would like this as well. 

user: frederikfab
issue_18913: state: open
title: [FEATURE] 2D roc_curve
body: In some cases, one wants to see the performance of applying two classifiers simultaneously. To see the performance one would want to look at a 2D roc curve, such as the one here https://cds.cern.ch/record/2703386/plots#0 

<img src="https://user-images.githubusercontent.com/13226500/100224703-76493b80-2f1d-11eb-8926-03e04a40558a.png" width="400" />

It would be very convenient to have this available as a canned method like the 1d roc curve. I would expect a signature like
```
tpr, fpr1, fpr2 = roc_curve2D(ypred1, ypred2, ytru)
```
where `tpr, fpr1, fpr2` would be `np.meshgrid`-like arrays

user: andrzejnovak
issue_18906: state: open
title: TST Test failures with OpenML for PyPy
body: The following tests are failing for `PyPy`:

```bash
FAILED datasets/tests/test_openml.py::test_fetch_openml_iris_pandas - TypeErr...
FAILED datasets/tests/test_openml.py::test_fetch_openml_iris_pandas_equal_to_no_frame
FAILED datasets/tests/test_openml.py::test_fetch_openml_iris_multitarget_pandas
FAILED datasets/tests/test_openml.py::test_fetch_openml_anneal_pandas - TypeE...
FAILED datasets/tests/test_openml.py::test_fetch_openml_cpu_pandas - TypeErro...
FAILED datasets/tests/test_openml.py::test_fetch_openml_as_frame_auto - TypeE...
FAILED datasets/tests/test_openml.py::test_convert_arff_data_dataframe_warning_low_memory_pandas
FAILED datasets/tests/test_openml.py::test_fetch_openml_adultcensus_pandas_return_X_y
FAILED datasets/tests/test_openml.py::test_fetch_openml_adultcensus_pandas - ...
FAILED datasets/tests/test_openml.py::test_fetch_openml_miceprotein_pandas - ...
FAILED datasets/tests/test_openml.py::test_fetch_openml_emotions_pandas - Typ...
FAILED datasets/tests/test_openml.py::test_fetch_openml_titanic_pandas - Type...
FAILED datasets/tests/test_openml.py::test_fetch_openml_verify_checksum[True]
```

with the following error traceback:

```bash
2020-11-24T16:19:24.2263007Z self = RangeIndex(start=0, stop=1, step=1)
2020-11-24T16:19:24.2263373Z 
2020-11-24T16:19:24.2263663Z     @cache_readonly
2020-11-24T16:19:24.2264148Z     def nbytes(self) -> int:
2020-11-24T16:19:24.2264461Z         """
2020-11-24T16:19:24.2264877Z         Return the number of bytes in the underlying data.
2020-11-24T16:19:24.2265280Z         """
2020-11-24T16:19:24.2265578Z         rng = self._range
2020-11-24T16:19:24.2265961Z >       return getsizeof(rng) + sum(
2020-11-24T16:19:24.2266404Z             getsizeof(getattr(rng, attr_name))
2020-11-24T16:19:24.2266869Z             for attr_name in ["start", "stop", "step"]
2020-11-24T16:19:24.2267216Z         )
2020-11-24T16:19:24.2267567Z E       TypeError: getsizeof(...)
2020-11-24T16:19:24.2268134Z E           getsizeof(object, default) -> int
2020-11-24T16:19:24.2268491Z E       
2020-11-24T16:19:24.2268855Z E           Return the size of object in bytes.
2020-11-24T16:19:24.2269205Z E       
2020-11-24T16:19:24.2269695Z E       sys.getsizeof(object, default) will always return default on PyPy, and
2020-11-24T16:19:24.2270318Z E       raise a TypeError if default is not provided.
2020-11-24T16:19:24.2270721Z E       
2020-11-24T16:19:24.2271186Z E       First note that the CPython documentation says that this function may
2020-11-24T16:19:24.2271859Z E       raise a TypeError, so if you are seeing it, it means that the program
2020-11-24T16:19:24.2272447Z E       you are using is not correctly handling this case.
2020-11-24T16:19:24.2272840Z E       
2020-11-24T16:19:24.2273307Z E       On PyPy, though, it always raises TypeError.  Before looking for
2020-11-24T16:19:24.2274004Z E       alternatives, please take a moment to read the following explanation as
2020-11-24T16:19:24.2274687Z E       to why it is the case.  What you are looking for may not be possible.
2020-11-24T16:19:24.2275112Z E       
2020-11-24T16:19:24.2275573Z E       A memory profiler using this function is most likely to give results
2020-11-24T16:19:24.2276218Z E       inconsistent with reality on PyPy.  It would be possible to have
2020-11-24T16:19:24.2276874Z E       sys.getsizeof() return a number (with enough work), but that may or
2020-11-24T16:19:24.2277656Z E       may not represent how much memory the object uses.  It doesn't even
2020-11-24T16:19:24.2278343Z E       make really sense to ask how much *one* object uses, in isolation
2020-11-24T16:19:24.2278956Z E       with the rest of the system.  For example, instances have maps,
2020-11-24T16:19:24.2279765Z E       which are often shared across many instances; in this case the maps
2020-11-24T16:19:24.2280459Z E       would probably be ignored by an implementation of sys.getsizeof(),
2020-11-24T16:19:24.2281121Z E       but their overhead is important in some cases if they are many
2020-11-24T16:19:24.2281768Z E       instances with unique maps.  Conversely, equal strings may share
2020-11-24T16:19:24.2282610Z E       their internal string data even if they are different objects---or
2020-11-24T16:19:24.2283287Z E       empty containers may share parts of their internals as long as they
2020-11-24T16:19:24.2284099Z E       are empty.  Even stranger, some lists create objects as you read
2020-11-24T16:19:24.2284800Z E       them; if you try to estimate the size in memory of range(10**6) as
2020-11-24T16:19:24.2285661Z E       the sum of all items' size, that operation will by itself create one
2020-11-24T16:19:24.2286395Z E       million integer objects that never existed in the first place.
```

I will open a PR marking them as `XFAIL` to not block #18879 but I think that we should investigate the underlaying issue.
user: alfaro96
issue_18893: state: open
title: Add native categorical support for linear models via coordinate descent solver
body: #### Describe the workflow you want to enable
I'd like to propose native categorical support for linear models, similar to #18394, i.e. ordinal encoded columns can be specified as "categorical". I see 3 main benefits:
- Possibly better user experience because one-hot-encoding becomes obsolete (though ordinal encoding is required as long `pandas.Categorical` is unsupported)
- Possible speed up of fitting as the orthogonal design of categoricals could be exploited.
- Possible better memory footprint in particular in combination with dense numerical features.

#### Describe your proposed solution
Add a new parameter `categorical_features` indicating which columns to treat as categoricals (ordinal encoded) as in #18394.

Then, add this functionality via a coordinate descent solver:
Internally, do as if categoricals were one-hot-encoded (as is done for the multiclass targets in HGBT, cf. [code here](https://github.com/scikit-learn/scikit-learn/blob/5fb02bb7bfed1a5edbe12ce942bb77cacde8180b/sklearn/ensemble/_hist_gradient_boosting/loss.py#L390)) and exploit the following structure:

For a feature (sub-) matrix `X` of a single one-hot-encoded feature and a diagonal weight matrix `W`, it holds that:
`X.t @ W @ X = diagonal`.

Thus, coordinate descent could loop in parallel over all levels/categories of this feature `X`, i.e. a parallelized block update.

#### Estimators
If only the existing coordinate descent solver is modified, then only squared error based estimators, i.e. `ElasticNet` and `Lasso`, would profit.
If a new or extended coordinate descent solver is OK, then several GLMs would also have native categorical support, i.e. `LogisticRegression`, `PoissonRegressor`, `TweedieRegressor`, etc. See also #16637.

user: lorentzenchr
issue_18889: state: open
title: [MRG] Add partial_fit function to DecisionTreeClassifier
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
First step for #18888 

#### What does this implement/fix? Explain your changes.
- Add the `partial_fit` function to `DecisionTreeClassifier`

#### Any other comments?
Collaboration of @neurodata

- Repo for ensemble attempts and benchmarks: https://github.com/PSSF23/SDTF/
- arXiv preprint: https://arxiv.org/abs/2110.08483

Thank you for feedback!
user: PSSF23
issue_18888: state: open
title: Add streaming option to decision trees
body: Collaborative effort of the @neurodata lab, under the directions of @jovo.

### Describe the workflow you want to enable

The current `sklearn.tree` module is designed for batch learning only and thus not capable of handling streaming data. I hope to make the tree builders capable of updating the existing tree and make streaming classifications possible.

### Describe your proposed solution

Stream decision tree and forest classifiers that use the `partial_fit` function to update existing tree structures. The function would build upon existing branches and incorporate new batches of data into the decision trees. Combined into simple [ensembles](https://github.com/PSSF23/SPDT/blob/main/spdt/forest.py), updated trees could achieve higher accuracy and produce consistent results.

We compared our algorithms with existing stream estimators: Hoeffding tree [[1](https://doi.org/10.1145/347090.347107)] from [`river`](https://github.com/online-ml/river) and Mondrian forest [[2](https://proceedings.neurips.cc/paper/5234-mondrian-forests-efficient-online-random-forests.pdf)] from [`scikit-garden`](https://github.com/scikit-garden/scikit-garden), in three classification tasks:

- [Splice Genes](https://archive.ics.uci.edu/ml/datasets/Molecular+Biology+(Splice-junction+Gene+Sequences))
- [Pen Digits](http://archive.ics.uci.edu/ml/datasets/pen-based+recognition+of+handwritten+digits)
- [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html)

Random forest and batch decision tree classifiers were also included. The benchmarks can be found [here](https://github.com/PSSF23/SPDT/tree/main/benchmarks).

Although our work was initially inspired by Hoeffding tree and Mondrian forest, they performed very inconsistently and caused severe memory issues. The disadvantages made the implementations unsuitable for high-dimensional classifications and large sample sizes. Thus, our simple approach turned out to be the optimal choice and could serve as a good starting point for exploring streaming trees.

#### First step - stream decision tree (#18889)
Implement the `partial_fit` function in `DecisionTreeClassifier` for modifying the existing tree with new batches of training samples. This step would check consistent shapes (features and classes) of training batches.

Potential changes needed on:
- [`sklearn/tree/_tree.pxd`](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/tree/_tree.pxd)
- [`sklearn/tree/_tree.pyx`](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/tree/_tree.pyx)
- [`sklearn/tree/_classes.py`](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/tree/_classes.py)

#### Second step - stream decision forest

Implement forest ensemble(s) that rely on the `partial_fit` function or add the function to existing classifiers like `RandomForestClassifier`.

Potential changes needed on:
- [`sklearn/ensemble/_forest.py`](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/ensemble/_forest.py)

#### Link for my fork & branch: https://github.com/PSSF23/scikit-learn-stream/tree/stream

### References
[1] Pedro Domingos and Geoff Hulten. 2000. Mining high-speed data streams. In Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining (KDD '00). Association for Computing Machinery, New York, NY, USA, 71–80. https://doi.org/10.1145/347090.347107

[2] Lakshminarayanan, Balaji, Daniel M. Roy, and Yee Whye Teh. "Mondrian forests: Efficient online random forests." Advances in neural information processing systems 27 (2014): 3140-3148.

user: PSSF23
issue_18882: state: open
title: CI failing because of fetch_openml(name='miceprotein', version=4)
body: The CI is failing everywhere, because the following is failing everywhere

```py
from sklearn.datasets import fetch_openml
fetch_openml(name='miceprotein', version=4)
```
even tho the dataset exist: https://www.openml.org/d/40966
user: thomasjpfan
issue_18860: state: open
title: [MRG] SimpleImputer: Handle string features where all values are missing
body: #### Reference Issues/PRs

#17526 - This is relevant, as I don't think it was possible to run into this issue before.

#### What does this implement/fix? Explain your changes.

When dealing with a sparse string feature, it's not unlikely that a particular CV split has all missing values for said feature. In cases like this, `SimpleImputer` appears to break. I believe this is because [`_validate_input` currently looks for a `str` value to determine `dtype`](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/impute/_base.py#L234-L236).

To me, it seems that if a user specifies a string for `fill_value`, that's another good indicator that `dtype` should be `object`.

#### Any other comments?

Thanks for your time and consideration!
user: ssaamm
issue_18859: state: open
title: memory Error / Agglomerative Clustering with Sparse Matrixes 
body: #### Describe the workflow you want to enable

Below is agglomerative clustering from scikit-learn. The problem is, to use the .fit method I need to convert the sparse matrix representation to an array. Since I have 80k documents with around 400k terms, the created array will have 80k x 400k elements. This will be a huge array and naturally, my computer has no memory and raised an memory Error/Exception.

`
from sklearn.cluster import AgglomerativeClustering

model = AgglomerativeClustering(n_clusters = n_clusters,
                                    linkage = linkage,
                                    affinity = affinity,
                                    distance_threshold = distance_threshold)
model = model.fit(vectorspace.toarray())
`

With KMeans, I don't need to transform the sparse matrix representation to an array. Notice that I don't need to call the .toarray() inside the model.fit(). Therefore, my computer does not raise an Exception, since it does not use that much memory.

`
model=KMeans(n_clusters=n_clusters)

model = model.fit(vectorspace)
`

Notice that most of the entries are 0 since most documents have only some overlapping terms, so a sparse matrix hugely reduces the memory used.
Does anyone know if there is already any other python module for agglomerative sparse matrixes?
Could you guys implement this 

#### Describe your proposed solution
Use the same algorithms with sparse matrixes. There are already defined multiplication/sum from other modules, so this shouldn't be very difficult. 

#### Describe alternatives you've considered, if relevant

#### Additional context
This is a huge issue with large collections/big data. I want to process a big collection in python, but I am not able do to HUGE memory requisites with dense matrixes.
user: esyker
issue_18858: state: open
title: Wrong shape of variance-covariance matrix in ARDRegressor when parameters are pruned
body: In the documentation for the ARDRegressor (https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ARDRegression.html), it says that the shape of `sigma_` is
```
sigma_ : array-like of shape (n_features, n_features)
```
However if parameters are pruned from the model the `sigma_` matrix will shrink, for example if 10 parameters are pruned then `sigma_` will be of shape `(n_features - 10, n_features - 10)`.

This could easily be fixed in the documentation, but it would of course be nice to have access to the full `sigma_` matrix.

Short example to demonstrate the behavior
``` python
import numpy as np
from sklearn.linear_model import ARDRegression

# make dataset
N = 100
M = 50
np.random.seed(42)
X = np.random.normal(0, 1, (N, M))
y = np.random.normal(0, 1, (N))

for lam in [1e7, 1e4]:
    ardr = ARDRegression(threshold_lambda=lam)
    ardr.fit(X, y)
    n_kept = np.count_nonzero(ardr.coef_)
    print('Nonzero parameters: {} / {}'.format(n_kept, M))
    print('sigma_ shape: ', ardr.sigma_.shape)
```
with output
```
Nonzero parameters: 50 / 50
sigma_ shape:  (50, 50)

Nonzero parameters: 37 / 50
sigma_ shape:  (37, 37)
```
user: erikfransson
issue_18854: state: open
title: support GPML minimize for sklearn.gaussian_process.GaussianProcessRegressor
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable

I'd like to add support for another internal optimizer for [sklearn.gaussian_process.GaussianProcessRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html). Currently, the only internal optimizer that is supported is "fmin_l_bfgs_b". I'm requesting support for Carl Rasmussen [minimize](http://learning.eng.cam.ac.uk/carl/code/minimize/) from GPML.

#### Describe your proposed solution

I don't think that the actual minimize function would be implemented in sklearn, but in scipy (please correct me if I'm wrong). So I can open up an issue on the scipy repository if this would be desirable in sklearn. This shouldn't break any existing code in sklearn as well since we can keep the "fmin_l_bfgs_b" default optimizer, but simply add support for a new one.

#### Describe alternatives you've considered, if relevant

I have found some python implementations of Carl Rasmussen's minimize online, although the function signature doesn't match exactly. I've yet to play around with it enough to get it to work, but that could be an alternative.

#### Additional context

A couple reasons I'm requesting this:

- I believe since [sklearn.gaussian_process.GaussianProcessRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html)'s implementation is from the GPML book, it makes sense to support the same minimize function that is used. It seems like most of the [Gaussian Process](https://scikit-learn.org/stable/modules/gaussian_process.html) module is implemented from GPML.

- I've personally had some issues trying to port some MATLAB code using GPML to sklearn and it _seems_ like it is an issue with the optimization step. I can use the optimized hyperparameters as fixed inputs in sklearn and achieve similar results without running any optimization. When running optimization, it seems like it produces worse fits (in my case)

#### Extras

- [GPML doc](http://www.gaussianprocess.org/gpml/code/matlab/doc/)
- [python impl of GPML minimize](https://github.com/marionmari/pyGPs/blob/master/pyGPs/Optimization/minimize.py) (BSD-2 license)

user: hkennyv
issue_18837: state: open
title: Duplicate check_finite when calling scipy.linalg functions
body: Most functions in `scipy.linalg` functions (e.g. `svd`, `qr`, `eig`, `eigh`, `pinv`, `pinv2` ...) have a default kwarg `check_finite=True` that we  typically leave to the default value in scikit-learn.

As we already validate the input data for most estimators in scikit-learn, this check is redundant and can cause significant overhead, especially at predict / transform time. We should probably always call those method with an explicit `check_finite=False` in scikit-learn.

This issue shall probably be addressed in many PRs, probably one per module that imports  `scipy.linalg`.

We should still make sure that the estimators raise a `ValueError` with the expected error message when fed with numpy arrays with infinite some values (`-np.inf`, `np.inf` or `np.nan`). This can be done manually by calling `sklearn.utils.estimator_checks.check_estimators_nan_inf` on the estimator, which should be automatically be called by `sklearn.tests.test_common` but we need to check that it's actually the case when reviewing such PRs.
user: ogrisel
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_18821: state: open
title: DOC add example to show how to deal with cross-validation
body: This new example shows which question cross-validation can answer.
It is also used to show how to inspect model parameters and hyperparameters and more importantly their variances.
user: glemaitre
issue_18817: state: open
title: Exponential gain function in dcg calculation
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
Please correct me if I wrong. 
Currently, the DCG score is [calculated](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf39bbdacd6ed713c00724f8f871d60370/sklearn/metrics/_ranking.py#L1057) with the linear gain function. 
```
    discount = 1 / (np.log(np.arange(y_true.shape[1]) + 2) / np.log(log_base))
    if k is not None:
        discount[k:] = 0
    if ignore_ties:
        ranking = np.argsort(y_score)[:, ::-1]
        ranked = y_true[np.arange(ranking.shape[0])[:, np.newaxis], ranking]
        cumulative_gains = discount.dot(ranked.T)
    else:
        discount_cumsum = np.cumsum(discount)
        cumulative_gains = [_tie_averaged_dcg(y_t, y_s, discount_cumsum)
                            for y_t, y_s in zip(y_true, y_score)]
        cumulative_gains = np.asarray(cumulative_gains)
    return cumulative_gains
```

However, in the industry, we use the alternative formulation of DCG as an exponential function, such as
` gains = 2 ** y_true - 1` 
The exp function emphasis retrieving the highly relevant docs.

Can we add the alternative formulation of gain?

user: zkid18
issue_18812: state: open
title: TfidfTransformer: idea to avoid unnecessary copy of csr matrices
body: #### Describe the workflow you want to enable

`TfidfTransformer.transform(X, copy=False)` shouldn't make copies of X but it does.
By _shouldn't_ I mean I didn't expect it to happen.

#### Describe your proposed solution

If both `X` and `self._idf_diag` are CSR matrices, one can do:

```python
for i in range(X.data.shape[0]):
    X.data[i] *= self._idf_diag.data[X.indices[i]]
X.eliminate_zeros()
```

...that is, if I'm not mistaken.

This can ofc be implemented efficiently in cython (there might be even some specialized function already?).

I will gladly submit a PR for this (currently I'm hacking around it) if somebody experienced in sklearn can guide me wrt where to put the code.
I never contributed to this project but would love to.

#### Describe alternatives you've considered, if relevant

#### Additional context

I'm working with big sparse matrices and this would help a lot.

See the comment at https://github.com/scikit-learn/scikit-learn/blob/05ce8141bc71ad21e55be4d1b3f6609f65e91e49/sklearn/feature_extraction/text.py#L1499

Line 1500 creates a copy.

EDIT: the `eliminate_zeros` is not even required I guess.
user: thebabush
issue_18798: state: open
title: Do we have a compelling reason to enforce tags?
body: With https://github.com/scikit-learn/scikit-learn/pull/18797 we are making tags an optional requirement for third-party estimators. If there was a compelling reason to adopt tags, I would think that third-party developers would adopt it.

Currently, the tags are use to hold the metadata of an estimator. This is mostly use in our estimator checks.

One of the selling points is that the tags can replace the attributes we have in our estimators. We recently deprecated `_pairwise` attribute in https://github.com/scikit-learn/scikit-learn/pull/18143 . We are also working on deprecating `_estimator_type` in https://github.com/scikit-learn/scikit-learn/pull/17806. In this point of view, the tags provide a namespace for provided metadata in estimators, which can be used by other libraries that ingest estimators.

Are there any other reasons for third party developers to adopt tags?
user: thomasjpfan
issue_18789: state: open
title: Proximity Matrix in RandomForestClassifier
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable

I think it would be great to formally implement the computation of a proximity matrix in the RandomForestClassifier class. 

#### Describe your proposed solution

I think the problem is pretty well-described. I propose implementing it as such: 

> The key output we want is the proximity (or similarity/dissimilarity) matrix. This is an n x n matrix where each value is the proportion of times observation i and j where in the same terminal node. For example, if 100 trees were fit and the ijth entry is 0.9, it means 90 times out of 100 observation i and j where in the same terminal node. 
-- http://gradientdescending.com/unsupervised-random-forest-example/

like

```
def proximity_matrix(X):
    apply(X)
    find proportion of indices that match between i and j
    compute the symmetric matrix
```

#### Describe alternatives you've considered, if relevant

I think this is easy to do outside of sklearn, but it seems to make sense to include it as a method in the class. 

#### Additional context

It looks like this has been discussed to various degrees on StackOverflow but hasn't been introduced yet. 

Mainly by @glouppe: [https://stackoverflow.com/questions/18703136/proximity-matrix-in-sklearn-ensemble-randomforestclassifier](https://stackoverflow.com/questions/18703136/proximity-matrix-in-sklearn-ensemble-randomforestclassifier)

It does seem to be something that is common in the main R implementation of RandomForests (https://stackoverflow.com/questions/23761942/proximity-matrix-random-forest-r)

I'm currently doing it like this, but I think it can be broadcasted to remove the double look: 

```
rfc = RandomForestClassifier()
rfc.fit(X)
tree_idxs = rfc.apply(X)
prox_mat = np.empty(shape=(n, n))

for i in range(n):
    for j in range(n):
        prox_mat[i,j] = np.sum(np.equal(tree_idxs[i,:], tree_idxs[j,:]))

return prox_mat 
```
user: vsocrates
issue_18786: state: open
title: [WIP] Add alpha parameter to GaussianProcessClassifier
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #18683

#### What does this implement/fix? Explain your changes.
Add an 'alpha' numerical stability term, similar to
GaussianProcessRegressor, to fix a reported issue caused by its absence.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: John-Ted
issue_18750: state: open
title: ENH allows checks generator to be pluggable
body: This PR allows passing a third-party generator that yields custom checks.

In imbalanced-learn, we reimplement the infrastructure developed in scikit-learn just to overwrite `_yield_all_checks` with our own generator. It would be more friendly to allow plugging any generator.

However, we should mention that the signature of the `check` functions can be changed at any time.
user: glemaitre
issue_18748: state: open
title: Support early_stopping with custom validation_set
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
Today in SGDClassifier, the parameter early_stopping uses a fraction of the data randomly, it would be useful to support a custom validation set chosen by the user. 

#### Describe your proposed solution
for example: 
```python
clf = SGDClassifier(early_stopping=True)
clf.fit(X_train, y_train, eval_set=(X_val, y_val))
```

EDIT
#### Broader Scope
Same applies to `GradientBoosting*` and `HistGradientBoosting*`

user: deltawi
issue_18738: state: open
title: ENH Extend plot_precision_recall_curve and plot_roc_curve to multiclass
body: closes #17170

**What does this implement/fix?** 
This PR extends the `plot_precision_recall_curve` and `plot_roc_curve` functions to accept multiclass scenarios. The plot can be either on the same figure if a single axis is passed or in n plots (where n=n_classes) by default.


user: claramatos
issue_18719: state: open
title: Precision-recall description improvement
body: #### Describe the issue linked to the documentation

<!--
Tell us about the confusion introduced in the documentation.
-->

https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html

It says, "high precision relates to a low false positive rate" and some a few places it links these two together, e.g., "false positives, decreasing precision."

#### Suggest a potential alternative/fix

<!--
Tell us how we could improve the documentation in this regard.
-->

"Precision = 1 - false discovery rate" and "Specificity = 1 - false positive rate"

Thus, the term "false discovery rate" should be emphasized, and "false positive rate" should be deemphasized when talking about high precision.
user: daniel-yj-yang
issue_18709: state: open
title: Issue with CalibratedClassifierCV with multiclass classification problems
body: While reviewing #17856, @ogrisel, @lucyleeow and myself found some weird things going on in the `CalibratedClassifierCV`. 

EDIT by Olivier: in particular we found out that our existing multiclass calibration test was very brittle and had a high likelhood of failing by changing the random seed.

The paper used as a reference is the following: 

Zadrozny, Bianca, and Charles Elkan. "Transforming classifier scores into accurate multiclass probability estimates." Proceedings of the Eighth ACM SIGKDD international conference on Knowledge discovery and data mining. 2002.

The issues are linked with the way to combine probabilities in multiclass settings.

### Issue with classifier natively supporting mutliclass problem

The paper mentioned that the multiclass problem should be tackled as a set of binary problems. However, classifiers that natively supports the multiclass problem, meaning without using one-vs-rest, will not be decoupled into one-vs-rest binary problems. Instead, we calibrated the different probabilities that the classifier output. So we don't implement what is written in the reference paper.

### Issue with the performance of one-vs-rest and normalization

The paper states 3 strategies to handle multiclass:

1. train one-vs-one binary classifiers and use "coupling" for merging binary probabilities;
2. train one-vs-one binary classifiers and use "least-squares" minimization for merging binary probabilities;
3. train one-vs-rest binary classifiers and normalize the probabilities to sum to 1.

We are implementing 3. However, we should revisit this approach with extensive testing and reproduce the experiment shown in the paper with the Brier score (MSE).
user: glemaitre
issue_18700: state: open
title: TransformerMixin as a base class
body: Is there a reason why the `TransformerMixin` class does not define the methods `fit` and `transform` even though the implemented `fit_transform` method references both of those methods?  It seems like bad practice to have a method in a base class reference functions that may or may not exist in children classes.  

I would imagine that as a base class it could implement both of those methods and have them raise `NotImplementedError` if its the responsibility of children classes.  This would improve editor compatibility - I have some methods that use type hints to define that a method is an `TransformerMixin` but I am getting linting warnings that `fit` and `transform` are not members of that class even though it seems that any child class must implement those methods.
user: apost71
issue_18694: state: open
title: Add option to inspection.permutation_importance to only compute importance for a subset of features
body: ## Describe the workflow you want to enable
We have datasets with very large numbers of features, where we only care about feature importance for a subset of them. This can happen when;
   - The model pipeline includes a transform in which only a subset of features are selected to pass on to the final estimator (eg via `ColumnTransformer`) and we know in advance that other columns can have no impact on the final model. 
   - We want to compute feature importance for different features at different stages of the transformation pipeline.

Currently permutation_importance computes always computes the importance for all features. This is slow and inefficient if you only want results for a small proportion of the features. 

#### Describe your proposed solution
Add an optional list of indices specifying the features for which to compute importance to eg `permutation_importance(...,features=None)`

Then adapt the function to iterate only through the specified indicies if supplied or all columns by default (as happens currently).
```
column_indexes = range(X.shape[1]) if features is None else features 
scores = Parallel(n_jobs=n_jobs)(delayed(_calculate_permutation_scores)(
        estimator, X, y, col_idx, random_seed, n_repeats, scorer
) for col_idx in column_indexes)
```



user: finnhacks42
issue_18683: state: open
title: Inconsistency between Gaussian Process Regressor vs Classifier causing LinAlgError
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
As my current dataset was encountering with LinAlgErrors ('n-th leading minor of the array is not positive definite') while being fitted with GaussianProcessClassifier, I delved into the code a little bit and saw that the classifier, contrary to its regressor counterpart, does not contain an alpha parameter to ensure numerical stability. What is missing is basically one line, namely [this one](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/gaussian_process/_gpr.py#L263) possibly just under [this one](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/gaussian_process/_gpc.py#L414). 

For a better understanding of the requirement here, see [here](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html#sklearn.gaussian_process.GaussianProcessRegressor) and [here](https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/gaussian_process/_gpr.py#L268).

#### Steps/Code to Reproduce
This is unfortunately not so easy, but one can try to reproduce this by using a dataset with feature count ~ sample count.

#### Versions
scikit-learn: 0.23.2


user: meakbiyik
issue_18680: state: open
title: post pruning tree
body: The existing cost-complexity based post pruning method of classification/regression trees in sklearn is not sufficient for my  research simulations. I would very much like a general post pruning method, where the user specifies a list of nodes which should be pruned (based on criteria that are "fluid", i.e. left to the user for experimentation purpose).

We tried to prune "manually" by simply setting the child pointers to -1, but that does not suffice for various other functions based on trees, see e.g.
https://github.com/scikit-learn/scikit-learn/issues/18468

Since there already exists a way to post prune trees in sklearn (based on cost-complexity), I am hoping that my request could be fulfilled by simply making that function public to the user?

Thanks!
Markus

user: markusloecher
issue_18679: state: open
title: [WIP] 'most_frequent' drop method for OneHotEncoder
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
#18553
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
Added `most frequent` option to `drop` argument of `OneHotEncoder`. Added attribute `categories_count_` to accomplish this. If feature categories count all equal, drops first category.

#### Any other comments?
Issue also asked for a `dropped_levels_` attribute, Let me know if this is something you would like added. 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: trewaite
issue_18676: state: open
title: Support roc_auc_score() for multi-class without probability estimates
body: I have a multi-class problem. I tried to calculate the ROC-AUC score using the function ```metrics.roc_auc_score()```. This function has support for multi-class but it needs the probability estimates, for that the classifier needs to have the method ```predict_proba()```. For example, ```svm.LinearSVC()``` does not have it and I have to use ```svm.SVC()``` but it takes so much time with big datasets.

Here is an example is an example of what I try to do:

```python
from sklearn import datasets
from sklearn.svm import SVC
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split


# Get the data
iris = datasets.load_iris()
X, y = iris.data, iris.target

# Create the model
clf = SVC(kernel='linear', probability=True)

# Split the data in train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)

# Train the model
clf.fit(X_train, y_train)

# Predict the test data
predicted = clf.predict(X_test)
predicted_proba = clf.predict_proba(X_test)
roc_auc = roc_auc_score(y_test, predicted_proba, multi_class='ovr')
```

If the classifier is changed to ```svm.LinearSVC()``` it will throw an error.  It will be useful to add support for multi-class problems without the probability estimates since ```svm.LinearSVC()``` is faster than ```svm.SVC()```.
user: luismiguells
labels: Needs Decision: Requires decision
issue_18671: state: open
title: LinearDiscriminantAnalysis does not handle degenerate problems well
body: As investigated when trying to solve unstable common tests in #18667, it seems that `LinearDiscriminantAnalysis` is bad a handling classification problems that are too trivial such as illustrated in the following (using MKL):

```python
>>> from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
>>> LinearDiscriminantAnalysis(solver="svd").fit([[0], [1], [1]], [0, 1, 1])
Traceback (most recent call last):
  File "<ipython-input-6-034ea174ac0e>", line 1, in <module>
    LinearDiscriminantAnalysis(solver="svd").fit([[0], [1], [1]], [0, 1, 1])
  File "/home/ogrisel/code/scikit-learn/sklearn/discriminant_analysis.py", line 553, in fit
    self._solve_svd(X, y)
  File "/home/ogrisel/code/scikit-learn/sklearn/discriminant_analysis.py", line 478, in _solve_svd
    _, S, Vt = linalg.svd(X, full_matrices=0)
  File "/home/ogrisel/miniforge3/envs/dev/lib/python3.8/site-packages/scipy/linalg/decomp_svd.py", line 121, in svd
    lwork = _compute_lwork(gesXd_lwork, a1.shape[0], a1.shape[1],
  File "/home/ogrisel/miniforge3/envs/dev/lib/python3.8/site-packages/scipy/linalg/lapack.py", line 950, in _compute_lwork
    ret = routine(*args, **kwargs)
ValueError: On entry to DGESDD parameter number 10 had an illegal value

>>> LinearDiscriminantAnalysis(solver="eigen").fit([[0], [1], [1]], [0, 1, 1])
/home/ogrisel/code/scikit-learn/sklearn/covariance/_empirical_covariance.py:88: UserWarning: Only one sample available. You may want to reshape your data array
  warnings.warn("Only one sample available. "
Traceback (most recent call last):
  File "<ipython-input-7-57ea859bdde2>", line 1, in <module>
    LinearDiscriminantAnalysis(solver="eigen").fit([[0], [1], [1]], [0, 1, 1])
  File "/home/ogrisel/code/scikit-learn/sklearn/discriminant_analysis.py", line 558, in fit
    self._solve_eigen(X, y,
  File "/home/ogrisel/code/scikit-learn/sklearn/discriminant_analysis.py", line 419, in _solve_eigen
    evals, evecs = linalg.eigh(Sb, Sw)
  File "/home/ogrisel/miniforge3/envs/dev/lib/python3.8/site-packages/scipy/linalg/decomp.py", line 578, in eigh
    raise LinAlgError('The leading minor of order {} of B is not '
LinAlgError: The leading minor of order 1 of B is not positive definite. The factorization of B could not be completed and no eigenvalues or eigenvectors were computed.

>>> lda = LinearDiscriminantAnalysis(solver="lsqr").fit([[0], [1], [1]], [0, 1, 1])
/home/ogrisel/code/scikit-learn/sklearn/covariance/_empirical_covariance.py:88: UserWarning: Only one sample available. You may want to reshape your data array
  warnings.warn("Only one sample available. "
LinearDiscriminantAnalysis(solver='lsqr')
```

The least squares solver does not fail (but still raises a warning) and the resulting model is unable to classifiy the training set correctly:

```python
>>> lda.predict([[0], [1]])
array([1, 1])
```

The above was done with scipy 1.5.2 with OpenBLAS from conda-forge under Linux.

I am not sure what we could do about it. Maybe we could at least try to raise the same exception for all the solvers when the problem is degenerate. The original exception of the eigen cause could still be raised as the cause of the user friendly exception (`raise e1 from e2` in Python 3). For the SVD case this happens when the estimated rank is 0.
user: ogrisel
issue_18669: state: open
title: Make TfidfVectorizer serializable in a reproducible way 
body: #### Describe the workflow you want to enable
- I would like to serialize (using `pickle` or `joblib`, as suggested in the [docs](https://scikit-learn.org/stable/modules/model_persistence.html)) a trained `TfidfVectorizer` object. Minimum working example:
```python
import joblib
from sklearn.feature_extraction.text import TfidfVectorizer, ENGLISH_STOP_WORDS

tfidf_model = TfidfVectorizer(stop_words=ENGLISH_STOP_WORDS)
tfidf_model.fit(["aaa", "ccc fff ", "ccc aaa bbb"])
joblib.dump(tfidf_model, "model.pkl")
```
- The problem is that if you run the code above twice, **the two resulting `model.pkl` binary files will differ.**
- As far as I understand, this is due to 2 different reasons.
   1. Some of the attributes of `tfidf_model` are not serialized in a deterministic way.
   2. Some of the attributes of `tfidf_model` are not defined in a deterministic way.
- **Why is this a relevant feature?** In my case I am using [`dvc`](https://dvc.org/) to track the results of my ML pipelines, and `dvc` relies on computing the MD5 hash of all the intermediate outputs to see if something has changed. So if running the same training script twice provides files with different MD5  hash, then I have a big issue because `dvc` thinks that the results have changed.

#### Describe your proposed solution
- A possible solution should address the 2 causes mentioned above.
- **Some of the attributes of `tfidf_model` are not serialized in a deterministic way.**
  - This is the case of the attributes of type `dict` (only for `python < 3.6` though) and of type `set`.
      - Use `collections.OrderedDict` instead of `dict` for the attribute`model.vocabulary_`.
      - Force the user to pass a `list` or `tuple` instead of a `set` for `stop_words` arguments in `TfidfVectorizer.__init__`. (Notice that the [docstring](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/text.py#L1593) already says `list`, but no type check is made and indeed `ENGLISH_STOP_WORDS` is a `set`)
- **Some of the attributes of `tfidf_model` are not defined in a deterministic way.**
  - This is the case of the attribute `_stop_words_id` which is [currently set to](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/text.py#L385) `self._stop_words_id = id(self.stop_words)`, therefore making it non reproducible across different runs. I do not really know how this one can be changed to fix the issue.
   

#### Describe alternatives you've considered, if relevant
Currently I am modifying the code above in the following hacky way to make it produce the same binary file, but I am not too proud of it.
```python
import joblib
from collections import OrderedDict
from sklearn.feature_extraction.text import TfidfVectorizer, ENGLISH_STOP_WORDS

tfidf_model = TfidfVectorizer(stop_words=sorted(ENGLISH_STOP_WORDS))
tfidf_model.fit(["aaa", "ccc fff ", "ccc aaa bbb"])
tfidf_model.vocabulary_ = OrderedDict(sorted(tfidf_model.vocabulary_.items(), key=lambda kv: kv[1]))
tfidf_model._stop_words_id = 0
joblib.dump(tfidf_model, "model.pkl")
```

#### Additional context
- I did not check it, but I think the same reasoning applies to `CountVectorizer` and `HashingVectorizer`.


user: FrancescoCasalegno
issue_18663: state: open
title: fit.transform not equal to fit_transform in NMF with solver 'mu' and init 'nndsvda'
body: #### Describe the bug
The test verifying that `fit.transform == fit_transform` failed for the `NMF` estimator with `solver='mu'` and init 'nndsvda'

Discovered in #16948, see also #18505.

cc @jeremiedbb @TomDLT , maybe @vene . Thanks!

#### Steps/Code to Reproduce
```python
from sklearn.utils.estimator_checks import check_transformer_general
from sklearn.decomposition import NMF

check_transformer_general('NMF', NMF(init='nndsvda', max_iter=1000, solver='mu'))
```

#### Expected Results
The test passes.

#### Actual Results
```pytr
Traceback (most recent call last):
  File "debug_nmf.py", line 4, in <module>
    check_transformer_general(
  File "/home/cmarmo/software/scikit-learn/sklearn/utils/_testing.py", line 302, in wrapper
    return fn(*args, **kwargs)
  File "/home/cmarmo/software/scikit-learn/sklearn/utils/estimator_checks.py", line 1305, in check_transformer_general
    _check_transformer(name, transformer, X, y)
  File "/home/cmarmo/software/scikit-learn/sklearn/utils/estimator_checks.py", line 1390, in _check_transformer
    assert_allclose_dense_sparse(
  File "/home/cmarmo/software/scikit-learn/sklearn/utils/_testing.py", line 409, in assert_allclose_dense_sparse
    assert_allclose(x, y, rtol=rtol, atol=atol, err_msg=err_msg)
  File "/home/cmarmo/.skldevenv/lib64/python3.8/site-packages/numpy/testing/_private/utils.py", line 1532, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/home/cmarmo/.skldevenv/lib64/python3.8/site-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0.01
fit_transform and transform outcomes not consistent in NMF(init='nndsvda', max_iter=1000, random_state=0, solver='mu')
Mismatched elements: 18 / 90 (20%)
Max absolute difference: 0.04219191
Max relative difference: 1.
 x: array([[5.584139e-01, 5.114790e-01, 4.818158e-01],
       [7.176271e-02, 9.486441e-02, 2.635424e-01],
       [1.848232e-01, 1.241254e-01, 9.756835e-22],...
 y: array([[5.566442e-01, 5.129683e-01, 4.835650e-01],
       [7.235885e-02, 9.501380e-02, 2.622291e-01],
       [1.846617e-01, 1.244062e-01, 1.105227e-07],...
```

#### Versions
```
System:
    python: 3.8.5 (default, Aug 12 2020, 00:00:00)  [GCC 10.2.1 20200723 (Red Hat 10.2.1-1)]
executable: /home/cmarmo/.skldevenv/bin/python
   machine: Linux-5.8.13-200.fc32.x86_64-x86_64-with-glibc2.2.5

Python dependencies:
          pip: 20.2.3
   setuptools: 41.6.0
      sklearn: 0.24.dev0
        numpy: 1.18.5
        scipy: 1.5.2
       Cython: 0.29.21
       pandas: 1.1.0
   matplotlib: 3.3.0
       joblib: 0.16.0
threadpoolctl: 2.1.0

Built with OpenMP: True
```
user: cmarmo
issue_18638: state: open
title: KBinsDiscretizer encodes wrong bin identifiers for quantile strategy
body: 
#### Describe the bug
When `KBinsDiscretizer` is used with `quantile` strategy to create ordinal encoding the created ID doesn't match the quantile identifier. Quantile ID should represent the quantile number where the encoded value reside. i.e. the encoding should output a meaningful value similar to the `uniform` strategy.

#### Steps/Code to Reproduce + Actual Results
Examples (which could later be used for testing):
```python
from sklearn.preprocessing import KBinsDiscretizer
import pandas as pd

encoder = KBinsDiscretizer(n_bins=3, encode='ordinal', strategy='quantile')
test_case_1 = [[0], [0], [0], [0], [3], [3]]
print(encoder.fit_transform(test_case_1).T) # [[0. 0. 0. 0. 1. 1.]]
test_case_2 = [[0], [0], [0], [3], [3], [3]]
print(encoder.fit_transform(test_case_2).T) # [[0. 0. 0. 0. 0. 0.]]
encoder = KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='quantile')
test_case_3 = [[0.05], [0.25], [0.25], [0.25], [0.45], [0.55], [0.65], [0.75], [0.75], [0.75]]
print(encoder.fit_transform(test_case_3).T) # [[0. 2. 2. 2. 3. 4. 5. 6. 6. 6.]]
test_case_35 = [[0.05], [0.24], [0.25], [0.25], [0.25], [0.45], [0.55], [0.65], [0.75], [0.75], [0.75]]
print(encoder.fit_transform(test_case_35).T) # [[0. 1. 2. 2. 2. 3. 4. 5. 5. 5. 5.]]
```

#### Expected Results
Quantile ID should be treated as an indicator of the quantile number which contain the value. Note that the inner implementation uses `np.percentile` so actually when the lowest value appear 2/3 of the times it could fall in quantile 6/10.
```python
from sklearn.preprocessing import KBinsDiscretizer
import pandas as pd

encoder = KBinsDiscretizer(n_bins=3, encode='ordinal', strategy='quantile')
test_case_1 = [[0], [0], [0], [0], [3], [3]]
print(encoder.fit_transform(test_case_1).T) # [[1. 1. 1. 1. 2. 2.]]
test_case_2 = [[0], [0], [0], [3], [3], [3]]
print(encoder.fit_transform(test_case_2).T) # [[1. 1. 1. 2. 2. 2.]]
encoder = KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='quantile')
test_case_3 = [[0.05], [0.25], [0.25], [0.25], [0.45], [0.55], [0.65], [0.75], [0.75], [0.75]]
print(encoder.fit_transform(test_case_3).T) # [[0. 3. 3. 3. 4. 5. 6. 9. 9. 9.]]
test_case_35 = [[0.05], [0.24], [0.25], [0.25], [0.25], [0.45], [0.55], [0.65], [0.75], [0.75], [0.75]]
print(encoder.fit_transform(test_case_35).T) # [[0. 1. 4. 4. 4. 5. 6. 7. 9. 9. 9.]]
```
Note that the maximal values get the highest quantile ID, and intermediate quantile IDs are not dense, which indicate a value which spans over multiple quantiles. 

#### Versions
Python dependencies:
          pip: 19.2.3
   setuptools: 41.4.0
      sklearn: 0.23.2
        numpy: 1.17.2
        scipy: 1.3.1
       Cython: 0.29.13
       pandas: 0.25.1
   matplotlib: 3.1.3
       joblib: 0.13.2
threadpoolctl: 2.1.0


Closing note: I see that quantile is deliberately implemented to use dense IDs, which I believe is confusing. I have the fix implemented and would gladly contribute it, of course if we decide it is the correct way.

user: antonpuz
labels: Needs Decision - API: Requires decision regarding API
issue_18632: state: open
title: [MRG] Fix svm gui readability
body: #### What does this implement/fix? Explain your changes.

1. The first commit changes the color of points with label 1 from white to blue so they can be visible (the background color being white too).

2. The second commit is to allow the user to zoom in on the plot with a button. In some cases after a fit the zoom goes back to a wide range (for example with only a few very close points). The "rectangular selection zoom" in the navigation bar creates a new point when clicking so it can't be used without changing the current configuration.

user: tanglef
issue_18630: state: open
title: PERF Cache intermediate results in decision tree estimators
body: I am wondering if it would be worth it to cache the intermediate results in decision tree estimators to avoid that the `sum_total` and `weighted_n_node_samples` attributes are re-computed.

In particular, these `for` loops will not be needed:

https://github.com/scikit-learn/scikit-learn/blob/6ca9eab67e1054a1f9508dfa286e0542c8bab5e3/sklearn/tree/_criterion.pyx#L330-L343

and:

https://github.com/scikit-learn/scikit-learn/blob/6ca9eab67e1054a1f9508dfa286e0542c8bab5e3/sklearn/tree/_criterion.pyx#L768-L780

and:

https://github.com/scikit-learn/scikit-learn/blob/6ca9eab67e1054a1f9508dfa286e0542c8bab5e3/sklearn/tree/_criterion.pyx#L1056-L1068

It should be easy to cache these intermediate results in `Stack` and `PriorityHeap`, but I think we should discuss before addressing the issue.
user: alfaro96
issue_18627: state: open
title: Contiguous feature importances permutation for a structured data matrix X
body: ## Describe the workflow you want to enable
In https://scikit-learn.org/stable/modules/generated/sklearn.inspection.permutation_importance.html#sklearn.inspection.permutation_importance, `permutation_importance` performs an evaluation of feature importances based on independent permutations of entries in the input X matrix.

If you have a structured dataset, say a multivariate time-series, or an image, then you inherently have correlated features either along the columns, rows, or both of the original data. If I want to feed that image into a Random Forest, then I would vectorize it and then construct my `X` matrix. Once you vectorize it to construct your data matrix, `X`, it will still have that inherent structure, but permutation importance will not make use of it.

I'm wondering if it is possible to implement a contiguous version of `permutation_importances` function where you can encode the structure of the contiguous chunks to take.

#### Describe your proposed solution
TBD.

#### Additional context
In https://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance_multicollinear.html#sphx-glr-auto-examples-inspection-plot-permutation-importance-multicollinear-py, there is a tutorial for handling this if the columns of `X` are all "independent" different features that are correlated, but you have to use a dendrogram.

user: adam2392
issue_18603: state: open
title: ENH: OOB Permutation Importance for Random Forests
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->
This implements out-of-bag permutation importance in random forests. This is why it provides value in addition to `inspection.permutation_importance`:

* It's standard in both main R Random Forest packages: `randomForest` and `Ranger`.
* OOB permutation importances do not require a significant change to the workflow (creating an additional validation set)
* Although it's covered in the documentation, many users likely do not realize just how biased impurity-based metrics are (see [here](https://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance.html#sphx-glr-auto-examples-inspection-plot-permutation-importance-py); it marks a random feature as most important).
  * For HistGradientBoostingClassifier/Regressor, impurity-based feature importances were left out entirely for this reason (see [#15132 (comment)](https://github.com/scikit-learn/scikit-learn/issues/15132#issuecomment-581253386))
* OOB permutation importances are faster and more stable than `inspection.permutation_importance` across repeated runs on the same data in the tests I've run (see [here](https://github.com/robert-robison/scikit-learn/blob/perm_imp_benchmarking/permutation_importance_benchmarking.ipynb))
* Would more seamlessly work with `SelectFromModel` (see discussion in #9606)

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
There are no open issues this directly addresses or closes to my knowledge. This has been discussed in multiple issues and PRs over the past several years, some of which are referenced above.

Issues/PRs that have suggested/implemented some type of RF permutation importance have been closed based on `eli5` permutation importance or, more recently, `sklearn.inspection.permutation_importance`:

* #3436
* #8027
* #8898
* #6024
* #16023, where @ogrisel states:

> As we have a model agnostic solution to assert feature importances without suffering from cardinality bias, I would be in favor of not extending our Random Forest implementation just to tackle bias issues in their feature importances, unless the method described in this paper is significantly better than permutation based feature importances but this would need to be demonstrated on some example.

I believe this is demonstrated by the reasons stated above, namely:

* Minimizes misinterpretation due to impurity-based importance
* Simplifies workflow by removing need for additional validation set
* Is more stable and faster

#### What does this implement/fix? Explain your changes.

The specific method that is implemented is the one shown in [Breimann (2001)](https://link.springer.com/content/pdf/10.1023/A:1010933404324.pdf), which is described in more details by the developers of the `Ranger` R package in the paper [Do Little Interactions get Lost in Dark Random Forests?](https://link.springer.com/article/10.1186/s12859-016-0995-8) The algorithm works like this:

* Get the OOB accuracy for each tree.
* Randomly shuffle a feature.
* Get the OOB accuracy again. The average drop in accuracy across all trees is the permutation importance for that feature.
* Repeat for all features.

For regression, the `r2_score` is used instead of accuracy.

This was implemented by adding an `importance_type` parameter that can take the values `{"impurity", "permutation"}`. While impurity-based importances aren't calculated until you need them, you need the dataset to calculate permutation importances, so they are calculated during the fit process if specified.

#### Any other comments?

The changes made here are functionally equivalent to the ones made in #3436, so I did reference that PR often while making these changes.

Also, this is my first pull request and this is kind of a big change, so I completely understand if I'm overstepping bounds here. Thank you!

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: robert-robison
labels: Needs Decision - API: Requires decision regarding API
issue_18555: state: open
title: API Deprecate using labels in bytes format
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #16980  

#### What does this implement/fix? Explain your changes.
Labels in bytes format are no supported for StratifiedKFold. But at present, the error message for the same is not helpful.

For instance, in the example below, scikit-learn is unable to catch the error properly and display an appropriate error message.
```python
import numpy as np
import pandas as pd
from sklearn.model_selection import StratifiedKFold
# generate data
nrows = 100
ncols = 3
X = np.random.rand(nrows,ncols)
y = pd.Series([b'a', b'b']*int(nrows/2))
# show y
print(y)
print(type(y[0]))
# split data
skf = StratifiedKFold(n_splits=2)
for train, test in skf.split(X, y):
    print(train, test)
```
Current error message:
```
ValueError: You appear to be using a legacy multi-label data representation. Sequence of sequences are no longer supported; use a binary array or sparse matrix instead - the MultiLabelBinarizer transformer can convert to this format.
```

In the above code, the error message does not make sense since the labels are not multi-label at all.
So I added a check that displays the following error message instead. 

```
ValueError: Labels represented as bytes is not supported. Convert the labels to a supported format. For example, y = y.astype(str)
```

#### Any other comments?
Perhaps we should add some tests for this as well in test_multilabel.py ?
If there is a better way to implement this, I am listening.
<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: cozek
labels: Needs Decision: Requires decision
issue_18553: state: open
title: add 'most_frequent' drop method to OneHotEncoder
body: I would like to propose adding a 'most_frequent' method as one of the `drop` parameter options in OneHotEncoder. 

I find that using the most frequent value as the reference level aids with interpreting the newly created OHE features. The 'first' method is not very intuitive. 

I would also be helpful if a `dropped_levels_` attribute was included, instead of having to derive it from the `categories_` and `drop_idx_` attributes. Thanks
user: kylegilde
issue_18552: state: open
title: add separator argument to OneHotEncoder get_feature_names method
body: I would like to be able to specify the separating characters between the feature name and feature value, so that I can reliably split on these separating characters to obtain the original feature name. 

Currently, a single underscore is hardcoded, but I would like to be able to specify a character sequence that is not likely to appear in the feature values, e.g. double underscores.

Thanks
user: kylegilde
issue_18550: state: open
title: Bug: VotingClassifier does not support estimators using class_weight
body: When using `VotingClassifier`, the estimators used inside cannot use the  `class_weight` parameter. The reason for that, I believe, is that `VotingClassifier` is encoding the labels before passing them to the single estimators, but not encoding the labels in `class_weight`. This throws a `ValueError: Class label <> not present`.

As a matter of fact, I wonder why this encoding would even happen in the first place, as it is handled by each model individually.

Here is a reproducible example:

```
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
import numpy as np

X = np.array([[1,2], [1,3], [2,1], [2,3], [2,3]])
y = np.array(["a", "b", "a", "b", "a"])

LR = LogisticRegression(class_weight={"a":1, "b": 2})
VC = VotingClassifier([("LR", LR)])

VC.fit(X, y)
```

Traceback:

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-3ae1f62ef011> in <module>
      9 VC = VotingClassifier([("LR", LR)])
     10 
---> 11 VC.fit(X, y)

/opt/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/_voting.py in fit(self, X, y, sample_weight)
    220         transformed_y = self.le_.transform(y)
    221 
--> 222         return super().fit(X, transformed_y, sample_weight)
    223 
    224     def predict(self, X):

/opt/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/_voting.py in fit(self, X, y, sample_weight)
     66                 delayed(_parallel_fit_estimator)(clone(clf), X, y,
     67                                                  sample_weight=sample_weight)
---> 68                 for clf in clfs if clf not in (None, 'drop')
     69             )
     70 

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in __call__(self, iterable)
   1002             # remaining jobs.
   1003             self._iterating = False
-> 1004             if self.dispatch_one_batch(iterator):
   1005                 self._iterating = self._original_iterator is not None
   1006 

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in dispatch_one_batch(self, iterator)
    833                 return False
    834             else:
--> 835                 self._dispatch(tasks)
    836                 return True
    837 

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in _dispatch(self, batch)
    752         with self._lock:
    753             job_idx = len(self._jobs)
--> 754             job = self._backend.apply_async(batch, callback=cb)
    755             # A job can complete so quickly than its callback is
    756             # called before we get here, causing self._jobs to

/opt/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py in apply_async(self, func, callback)
    207     def apply_async(self, func, callback=None):
    208         """Schedule a func to be run"""
--> 209         result = ImmediateResult(func)
    210         if callback:
    211             callback(result)

/opt/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py in __init__(self, batch)
    588         # Don't delay the application, to avoid keeping the input
    589         # arguments in memory
--> 590         self.results = batch()
    591 
    592     def get(self):

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in __call__(self)
    254         with parallel_backend(self._backend, n_jobs=self._n_jobs):
    255             return [func(*args, **kwargs)
--> 256                     for func, args, kwargs in self.items]
    257 
    258     def __len__(self):

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in <listcomp>(.0)
    254         with parallel_backend(self._backend, n_jobs=self._n_jobs):
    255             return [func(*args, **kwargs)
--> 256                     for func, args, kwargs in self.items]
    257 
    258     def __len__(self):

/opt/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/_base.py in _parallel_fit_estimator(estimator, X, y, sample_weight)
     34             raise
     35     else:
---> 36         estimator.fit(X, y)
     37     return estimator
     38 

/opt/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py in fit(self, X, y, sample_weight)
   1599                       penalty=penalty, max_squared_sum=max_squared_sum,
   1600                       sample_weight=sample_weight)
-> 1601             for class_, warm_start_coef_ in zip(classes_, warm_start_coef))
   1602 
   1603         fold_coefs_, _, n_iter_ = zip(*fold_coefs_)

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in __call__(self, iterable)
   1002             # remaining jobs.
   1003             self._iterating = False
-> 1004             if self.dispatch_one_batch(iterator):
   1005                 self._iterating = self._original_iterator is not None
   1006 

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in dispatch_one_batch(self, iterator)
    833                 return False
    834             else:
--> 835                 self._dispatch(tasks)
    836                 return True
    837 

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in _dispatch(self, batch)
    752         with self._lock:
    753             job_idx = len(self._jobs)
--> 754             job = self._backend.apply_async(batch, callback=cb)
    755             # A job can complete so quickly than its callback is
    756             # called before we get here, causing self._jobs to

/opt/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py in apply_async(self, func, callback)
    207     def apply_async(self, func, callback=None):
    208         """Schedule a func to be run"""
--> 209         result = ImmediateResult(func)
    210         if callback:
    211             callback(result)

/opt/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py in __init__(self, batch)
    588         # Don't delay the application, to avoid keeping the input
    589         # arguments in memory
--> 590         self.results = batch()
    591 
    592     def get(self):

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in __call__(self)
    254         with parallel_backend(self._backend, n_jobs=self._n_jobs):
    255             return [func(*args, **kwargs)
--> 256                     for func, args, kwargs in self.items]
    257 
    258     def __len__(self):

/opt/anaconda3/lib/python3.7/site-packages/joblib/parallel.py in <listcomp>(.0)
    254         with parallel_backend(self._backend, n_jobs=self._n_jobs):
    255             return [func(*args, **kwargs)
--> 256                     for func, args, kwargs in self.items]
    257 
    258     def __len__(self):

/opt/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py in _logistic_regression_path(X, y, pos_class, Cs, fit_intercept, max_iter, tol, verbose, solver, coef, class_weight, dual, penalty, intercept_scaling, multi_class, random_state, check_input, max_squared_sum, sample_weight, l1_ratio)
    841     le = LabelEncoder()
    842     if isinstance(class_weight, dict) or multi_class == 'multinomial':
--> 843         class_weight_ = compute_class_weight(class_weight, classes, y)
    844         sample_weight *= class_weight_[le.fit_transform(y)]
    845 

/opt/anaconda3/lib/python3.7/site-packages/sklearn/utils/class_weight.py in compute_class_weight(class_weight, classes, y)
     63             i = np.searchsorted(classes, c)
     64             if i >= len(classes) or classes[i] != c:
---> 65                 raise ValueError("Class label {} not present.".format(c))
     66             else:
     67                 weight[i] = class_weight[c]

ValueError: Class label a not present.
```

This issue was noticed [here](https://stackoverflow.com/questions/60992466/using-class-weights-with-sklearn-votingclassifier#comment108421729_60992466).

System:
    python: 3.7.6 
   machine: macOS

   sklearn: 0.22.1
     numpy: 1.18.1
     scipy: 1.4.1
user: MaximeKan
issue_18542: state: open
title: Adding sequential Information Bottleneck (sIB)
body: Hi,

I'm Assaf from IBM Research. We recently open-sourced our new implementation of the clustering algorithm, sequential information Bottleneck (sIB), and I thought that it could be a useful addition to SciKit-Learn. 

The repository is [this](https://github.com/IBM/sib) and a link to the academic paper describing the algorithm can be found [here](https://dl.acm.org/doi/abs/10.1145/564376.564401) (published in SIGIR 2002, has more than 300 citations). We also prepared a [document](https://github.com/IBM/sib/blob/master/docs/sib_implementation.pdf) giving background information on sIB and detailing the implementation of partition optimization, which is the core of the algorithm.

I'm the main developer and the maintainer of this code-base and I have time allocation for integration work and maintenance work on sIB in SciKit-Learn, if the community is interested in that. I also have time allocation for further improvements as detailed below.

In what follows I'll try to highlight some of the features that make sIB different than more common clustering algorithms, and to provide details about its new implementation. 

Thank you


## About sIB
(text copied from [here](https://github.com/IBM/sib/blob/master/docs/sib_implementation.pdf))

In contrast to algorithms such as K-Means, where the objective function and the distance function that they employ are two separate entities, in the case of sIB, the objective of in-cluster information maximization dictates the formula of the distance function by means of a mathematical derivation. Thus, sIB benefits from using an optimal distance function for the objective it uses, which turns out to be a weighted Jensen-Shannon divergence. However, this distance function is also more compute-intensive compared to distance functions such as Euclidean or Cosine distance.

Moreover, sIB is a sequential algorithm, which means that (a) before computing the new cluster for a sample, sIB withdraws the sample from its current cluster to prevent that sample from biasing the distance function towards keeping it in the same cluster, and (b) sIB updates the centroids while iterating over the samples and not only at the end of the iteration. 

Overall, while iterating over the samples, every sample is withdrawn from its cluster, the centroid of that cluster is updated, a new cluster is selected for the sample using the weighted JS divergence distance function, then the sample is added to the new cluster and eventually the centroid of the new cluster is updated. In total, sIB performs _**2**_*_**n_samples**_ centroid updates during a full iteration, while Lloyd's K-Means performs only _**n_clusters**_ updates.

By taking into account both the higher computational intensity of sIB's distance function and the significantly higher number of centroid updates that it performs, it is easy to see why sIB is inherently slower than algorithms such as Lloyd K-Means. However, as we show here, an efficient implementation is employable and closes the run-time gap between sIB and K-Means to a large extent.

## Comparison to Scikit-Learn's Lloyd K-Means
I conducted a set of experiments using the 20 News Groups dataset and used standard clustering metrics to evaluate results. A heatmap is also used to visualize the quality of the clustering. In pretty much all setups, sIB comes out significantly better than K-Means in terms of quality and also about 2 times slower: For illustration, this is the output from _Example2_ in the [examples directory](https://github.com/IBM/sib/tree/master/examples):

### Hardware
Lenovo P50 with Intel i7 @ 2.7Ghz, 4 cores, 8 with hyper-threading:

### Stats 
KMeans(algorithm='full', max_iter=15, n_clusters=20, n_jobs=-1, n_init=10)
Size: 18846 vectors
Time: 26.850 seconds
Measures:
	Adjusted Mutual Information: 0.228
	Adjusted Rand Index: 0.071

sIB(n_cluseters=20, n_jobs=-1, n_init=10, max_iter=15, tol=0.02, random_state=None, uniform_prior=True, inv_beta=0, optimizer_type=C, verbose=False)
Size: 18846 vectors
Time: 64.555 seconds
Measures:
	Adjusted Mutual Information: 0.595
	Adjusted Rand Index: 0.466

### Heat-maps
The heat-maps below show the distribution of classes over clusters. If we have a perfect clustering analysis, then there will be a 1:1 match between a class and a cluster, and the heat-map will look diagonal. The more diagonal the heat-map, the more accurate the clustering analysis is. The Y-axis shows the original classes from the dataset, the X-axis shows the cluster ids.

![kmeans_heatmap](https://user-images.githubusercontent.com/28494432/95176223-3db29e80-07c5-11eb-829d-c9fd2e177fcf.png)

![sib_heatmap](https://user-images.githubusercontent.com/28494432/95176245-460ad980-07c5-11eb-9a46-d6a60dd3876a.png)

### Notes on the comparison
We can see that sIB takes  2.4 times longer than K-Means; its AMI score is 2.6 times higher than K-Means, and its ARI score is 6.5 times better than K-Means. Note that running K-Means with more restarts and more iterations doesn't help it to get closer to sIB's scores. 

## About the Implementation
The code is organized according to SciKit-Learn's API with the common methods of fit()/transform()/etc. 

The algorithm is divided into two parts - 
1. The main workflow, implemented in Python in the file sib.py. This includes the data preparation, parallel execution of initializations, aggregation of results and selection of the best init. 
2. Partition optimization, implemented twice - once in pure Python in PSibOptimizer.py and once in C++ wrapped by Cython in CSibOptimizer.py and in the c_package directory. Both the pure Python and the C++ implementations yield equal results and the choice between them is left to the user (the default is the C++). Both take about 150 lines of code. The C++ code, despite being a vanilla C++ without any vectorization or parallelizm at the mathematical level, and without using any explicit caching, is twice as fast as the Python code. The Python code relies on Numpy's log and einsum functions over large matrices and I consider it efficient. It also aims to cache log computations and re-use memory. I believe that both implementations can be improved with the use of BLAS but I'm not an expert in this domain. Help will be very appreciated. Adding a Cython implementation is also an option and could benefit from both worlds of direct access to memory and easier way to BLAS. If C++ code is less desired in SciKit-Learn, it is possible to stick to the pure Python implementation until a Cython version will be ready. 

The algorithm supports both Sparse (csr_matrix) and Dense (ndarray) vector representations. In addition, it supports both integer and floating-point data types. In the python optimizer this is supported out of the box, in this C++ code this is supported by a template. In any case it is transparent to the user. The integer representation is preferable as integral math is faster than floating-point math, and it is also easier to optimize by means of a lookup table. See more details about this in the implementation document, at the end. 


## Future Plans
1. Switch from joblib to OpenMP
2. Investigate the use of a lookup table to avoid most computations of log(x) in run-time. Preliminary experiments showed substantial improvement. 
3. Investigate the benefits of a Cython implementation and perhaps switch from C++ to it.
4. If the C++ is integrated, look into vectorization and parallelizm. 





user: assaftibm
issue_18540: state: open
title: RandomForestRegressor quantile Criterion
body: For `sklearn` **version 0.23.2**,  the parameter `criterion` of `RandomForestRegressor`  only supports `mse` and `mae`. Is there any plan to extend these two options for a general quantile (e.g. 95% quantile, 25% quantile ... etc), such that a prediction interval can be made for random forest regression ?
user: newTypeGeek
issue_18531: state: open
title: Ward linkage ValueError with l2 affinity (Agglomerative Clustering)
body: 
#### Describe the bug
When using sklearn.cluster.AgglomerativeClustering, a ValueError "Ward can only work with euclidean distances." is thrown when using ward linkage and l2 affinity. l2 should **also** work since it is the same thing as euclidean. This is a quite simple problem and I have a fix ready to submit a pull request.

#### Steps/Code to Reproduce
To simply reproduce with no imports. (self.linkage and self.affinity are altered here to make it easily reproducible)

```python
self_linkage = 'ward'
self_affinity = 'l2'     
        
if self_linkage == "ward" and self_affinity != "euclidean":
  raise ValueError("%s was provided as affinity. Ward can only "
  "work with euclidean distances." %
  (self_affinity, ))

```

#### Expected Results
When using l2 affinity, no ValueError should be thrown.

#### Actual Results
ValueError("%s was provided as affinity. Ward can only "
  "work with euclidean distances." %
  (self_affinity, ))

#### Versions
System:
    python: 3.6.9 (default, Jul 17 2020, 12:50:27)  [GCC 8.4.0]
executable: /usr/bin/python3
   machine: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
       pip: 19.3.1
setuptools: 50.3.0
   sklearn: 0.22.2.post1
     numpy: 1.18.5
     scipy: 1.4.1
    Cython: 0.29.21
    pandas: 1.1.2
matplotlib: 3.2.2
    joblib: 0.16.0


<!-- Thanks for contributing! -->

user: mas1
issue_18530: state: open
title: Are predictions on scikit-learn models thread-safe?
body: Given some classifier (SVC/Forest/NN/whatever) is it safe to call `.predict` on the same instance concurrently from different threads?

From a distant point of view, my guess is they do not mutate any internal state. But I did not find anything in the docs about it.

Here is a minimal example showing what I mean:

```python3
#!/usr/bin/env python3
import threading

from sklearn import datasets
from sklearn import svm
from sklearn.ensemble import RandomForestClassifier
from sklearn.neural_network import MLPClassifier

X, y = datasets.load_iris(return_X_y=True)

# Some model. Might be any type, e.g.:
clf = svm.SVC()
clf = RandomForestClassifier(),
clf = MLPClassifier(solver='lbfgs')

clf.fit(X, y)


def use_model_for_predictions():
    for _ in range(10000):
        clf.predict(X[0:1])


# Is this safe?
thread_1 = threading.Thread(target=use_model_for_predictions)
thread_2 = threading.Thread(target=use_model_for_predictions)
thread_1.start()
thread_2.start()
```
user: Dobiasd
labels: module:test-suite: everything related to our tests
issue_18521: state: open
title: [MRG] ComplementNB with class priors when computing joint log-likelihoods (Trac : #14523, #14444)
body: #### Reference Issues/PRs

Fixes issue : #14444
Follows PR : #14523


#### What does this implement/fix? Explain your changes.

Permits the use of the priors in the ComplementNB by adding the possibility to chose if we need them or not.
(`if self.fit_prior == True`)
user: yacth
issue_18507: state: open
title: Save training and vaildation loss in `loss_curve_` in MLPClassifier and MLPRegressor with early_stopping
body: Let's select MLPClassifier.

In MLPClassifier there is `loss_curve_` available. If there is `early_stopping` enabled then some part of the data is used as validation. Can we save the loss of training and validation data in the `loss_curve_` as well?

#### Additional context

I've compared the MLP implementation with Tensorflow implementation and it works very well, there are no significant differences in the performance. You can read the comparison details at [my blog post](https://mljar.com/blog/tensorflow-vs-scikit-learn/). I'm using the MLP in my AutoML [**mljar-supervised**](https://github.com/mljar/mljar-supervised) which creates Markdown reports for each model. I would like to have learning curves available in the report.
user: pplonski
issue_18497: state: open
title: Extend binary clf metrics (precision recall, roc, etc...) to handle nuances of detection problems
body: ### Describe the workflow you want to enable

The current implementation of `_binary_clf_curve` provides the backbone for several common metrics, namely PR and ROC-curves. In its current form it works correctly for classification problems, but unfortunately it is not amenable to detection problems. This is partially due to the open-world nature of detection problems, yet people still are able to compute PR and ROC curves for these problems. However, none of these curves are ever computed using the stable and ubiquitous scikit-learn API, and I think that's a problem because it leads to inconsistencies between scoring methods. 

It took me awhile but I believe I understand the essence of why scikit-learn cannot be used for these problems, and I also believe I have a simple fix that would enable more widespread adoption. 

In principle it should be possible to coerce an open-world detection problem into a closed-world classification problem. All object detection scoring methods (COCO, VOC, etc...) effectively do this. For each image they have a set of true detections and a set of predicted detections. Using some heuristic they assign true detections to predicted detections. Now that we have a matching, we can build out y_true, y_pred, and y_score lists. 

If a predicted box matches a true box, append the their classes to y_true, y_pred, and the score to y_score (better yet remember the score of each category). 

If a predicted box is unmatched, and its score is above a threshold it will be a false positive, so we can add a -1 to y_true, the predicted class to y_pred, and the score to y_score. 

The last case is where the issue arises. If a true box is unmatched by predicted boxes, we can place the true class in y_true, a -1 in y_pred, and now we need to put a score into y_score. For a long time I was just putting in a zero. 

However, that leads to a problem. When the threshold is 0 that will give you 100% recall, even on the true boxes where no predicted boxes were detected. This leads to the essence of this proposal: 


### Describe your proposed solution

**Scikit-learn needs a way to specify a true class in the `y_true`, `y_pred`, `y_score` lists that will always result in a false negative in the result of `_binary_clf_curve`**. This would effectively encode a true object that will never be able to be a true positive (because the detector simply failed to predict a box around it at all threshold levels). 

I think the easiest way to encode this would be to allow `y_score` to contain a `nan` value, which indicates no score has been assigned to this truth. If `_binary_clf_curve` could handle this (it currently raises an error), then it should solve the problem. 

### Additional context

Scoring code that I use to coerce detection problems into scikit-learn style "Confusion Vectors": https://github.com/Kitware/kwcoco/blob/master/kwcoco/metrics/confusion_vectors.py#L29

---- 

EDIT:

Instead of `np.nan` it might be better to use `-np.inf` as the score because it makes the rest of the code work without much modification (although I do think consumers of `_binary_clf_curve` might need to adapt to this case). 

```python

def _binary_clf_curve(y_true, y_score, pos_label=None, sample_weight=None):
    """
    MODIFIED VERSION OF SCIKIT-LEARN API

    Calculate true and false positives per binary classification threshold.

    Parameters
    ----------
    y_true : array, shape = [n_samples]
        True targets of binary classification

    y_score : array, shape = [n_samples]
        Estimated probabilities or decision function

    pos_label : int or str, default=None
        The label of the positive class

    sample_weight : array-like of shape (n_samples,), default=None
        Sample weights.

    Returns
    -------
    fps : array, shape = [n_thresholds]
        A count of false positives, at index i being the number of negative
        samples assigned a score >= thresholds[i]. The total number of
        negative samples is equal to fps[-1] (thus true negatives are given by
        fps[-1] - fps).

    tps : array, shape = [n_thresholds <= len(np.unique(y_score))]
        An increasing count of true positives, at index i being the number
        of positive samples assigned a score >= thresholds[i]. The total
        number of positive samples is equal to tps[-1] (thus false negatives
        are given by tps[-1] - tps).

    thresholds : array, shape = [n_thresholds]
        Decreasing score values.

    Example
    -------
    >>> y_true  = [      1,   1,   1,   1,   1,   1,   0]
    >>> y_score = [ np.nan, 0.2, 0.3, 0.4, 0.5, 0.6, 0.3]
    >>> sample_weight = None
    >>> pos_label = None
    >>> fps, tps, thresholds = _binary_clf_curve(y_true, y_score)
    """
    import numpy as np
    from sklearn.utils import assert_all_finite
    from sklearn.utils import column_or_1d
    from sklearn.utils import check_consistent_length
    from sklearn.utils.multiclass import type_of_target
    from sklearn.utils.extmath import stable_cumsum
    # Check to make sure y_true is valid
    y_type = type_of_target(y_true)
    if not (y_type == "binary" or
            (y_type == "multiclass" and pos_label is not None)):
        raise ValueError("{0} format is not supported".format(y_type))

    check_consistent_length(y_true, y_score, sample_weight)
    y_true = column_or_1d(y_true)
    y_score = column_or_1d(y_score)
    assert_all_finite(y_true)
    # assert_all_finite(y_score)

    if sample_weight is not None:
        sample_weight = column_or_1d(sample_weight)

    # ensure binary classification if pos_label is not specified
    # classes.dtype.kind in ('O', 'U', 'S') is required to avoid
    # triggering a FutureWarning by calling np.array_equal(a, b)
    # when elements in the two arrays are not comparable.
    classes = np.unique(y_true)
    if (pos_label is None and (
            classes.dtype.kind in ('O', 'U', 'S') or
            not (np.array_equal(classes, [0, 1]) or
                 np.array_equal(classes, [-1, 1]) or
                 np.array_equal(classes, [0]) or
                 np.array_equal(classes, [-1]) or
                 np.array_equal(classes, [1])))):
        classes_repr = ", ".join(repr(c) for c in classes)
        raise ValueError("y_true takes value in {{{classes_repr}}} and "
                         "pos_label is not specified: either make y_true "
                         "take value in {{0, 1}} or {{-1, 1}} or "
                         "pass pos_label explicitly.".format(
                             classes_repr=classes_repr))
    elif pos_label is None:
        pos_label = 1.

    # make y_true a boolean vector
    y_true = (y_true == pos_label)

    # Transform nans into negative infinity
    nan_flags = np.isnan(y_score)
    y_score[nan_flags] = -np.inf

    # sort scores and corresponding truth values
    desc_score_indices = np.argsort(y_score, kind="mergesort")[::-1]
    y_score = y_score[desc_score_indices]
    y_true = y_true[desc_score_indices]
    if sample_weight is not None:
        weight = sample_weight[desc_score_indices]
    else:
        weight = 1.

    # y_score typically has many tied values. Here we extract
    # the indices associated with the distinct values. We also
    # concatenate a value for the end of the curve.

    with np.errstate(invalid="ignore"):
        y_diff = np.diff(y_score)
    # Set difference between -inf to zero
    fix_flags = np.isinf(y_score[:-1]) & np.isnan(y_diff)
    y_diff[fix_flags] = 0

    distinct_value_indices = np.where(y_diff)[0]
    threshold_idxs = np.r_[distinct_value_indices, y_true.size - 1]

    # accumulate the true positives with decreasing threshold
    tps = stable_cumsum(y_true * weight)[threshold_idxs]
    if sample_weight is not None:
        # express fps as a cumsum to ensure fps is increasing even in
        # the presence of floating point errors
        fps = stable_cumsum((1 - y_true) * weight)[threshold_idxs]
    else:
        fps = 1 + threshold_idxs - tps
    return fps, tps, y_score[threshold_idxs]

```
user: Erotemic
issue_18492: state: open
title: [WIP] online matrix factorization with missing values
body: #### What does this implement/fix? Explain your changes.
Allow dict_learning_online to handles nan 

todo 

- [ ] add tests
- [x] add minibatch
- [ ] documentation
- [ ] support custom missing value marker
- [ ] partial fit
- [ ] transform
- [ ] write an example

#### Any other comments?
Adapted from  "Online Group-Structured  Dictionary Learning " by Z. Szabo, B. Poczos, A. Lörnicz, 2011

The benchmarks are as follow: 
The execution time of dict_learning_na is similar on small dataset w.r.t. dict_learning_online
![time](https://user-images.githubusercontent.com/9698757/94562563-c54c5a80-0265-11eb-9c59-3972922c8cb1.png)

The loss is also comparable to dict_learning_online
![loss](https://user-images.githubusercontent.com/9698757/94562688-ed3bbe00-0265-11eb-912c-decfe4bb0b4a.png)

The reconstruction of missing values is compared with KNN, SoftImpute and IterativeSVD from fancyimpute. On the digits dataset
![digits_reconstruction](https://user-images.githubusercontent.com/9698757/94563014-62a78e80-0266-11eb-982e-3cff5577007d.png)

user: TwsThomas
issue_18479: state: open
title: FEA Add cumulative gain curve metric
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Closes #10003


#### What does this implement/fix? Explain your changes.
Add a new metric for binary classification known as cumulative_gain_curve

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Alit10
issue_18462: state: open
title: make_ranking generated datasets function
body: 
#### Describe the workflow you want to enable
Hello, a `make_ranking` function would be useful to me, to generate data for input into a learning-to-rank estimator (e.g. LightGBM's [LGBMRanker).](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRanker.html) There are currently many `make_*` dataset generation functions, but none for LETOR objectives. Where as classification and regression have three main components - X (feature data), y (dependent variable/labels), and sample weights (optional), learning to rank comes with a fourth component, a group vector indicating how samples are nested within groups. In IR settings, the group is usually the query and the samples are documents returned by a search engine for that query. Also, there are idiosyncrasies in ranking data like "position bias" where the top item in a group is more likely to have a positive label than items further down the list.

#### Describe your proposed solution
I would add a `make_ranking` function in `datasets._samples_generator` (and applicable tests). The function would return X, y, and now a group size vector. It would support generating both fixed group sizes and random group sizes that hover around an average group size, allowing the user to directly input group sizes, and a way to bias relevance towards the front of each group.

#### Describe alternatives you've considered, if relevant
I can write my own data generators and host them in gists, not really relevant though.

#### Additional context
Here's a proposed implementation:

```
def make_ranking(n_groups=100, n_per_group=10, n_features=2, *, gmax=1
                 , pos_bias=False, random_group_sizes=False, random_state=42):

    if gmax < 1:
        raise ValueError('maximum relevance grade (gmax) should be >= 1')

    generator = check_random_state(random_state)

    # -- when n_groups is a vector, it is a vector of group sizes.
    if hasattr(n_groups, "__len__"):
        group_sizes = n_groups
    else:
        if random_group_sizes:
            group_sizes = np.maximum(generator.binomial(2 * n_per_group, p=0.5, size=n_groups), 1)

        else:
            group_sizes = np.array([n_per_group] * n_groups)

    n_samples = np.sum(group_sizes)

    # -- features come from a multivariate standard normal distribution
    X = generator.multivariate_normal(np.zeros(n_features)
                                      , cov=np.identity(n_features)
                                      , size=(n_samples,))

    # -- create grouped relevance labels
    y = np.zeros(n_samples, dtype=int)
    y_idx = 0
    for gs in group_sizes:
        rel_probs = generator.uniform(size=gs)

        if pos_bias:
            rel_probs = np.sort(rel_probs)[::-1]

        is_rel_vec = rel_probs >= generator.uniform(size=gs)
        n_rel_samples = int(np.sum(is_rel_vec))

        if n_rel_samples:
            rel_scores = generator.choice(range(gmax + 1), size=n_rel_samples)
            y[y_idx + np.where(is_rel_vec)[0]] = rel_scores

        y_idx += gs

    return X, y, group_sizes
```

user: ffineis
issue_18461: state: open
title: Error on using None for missing values in SimpleImputer with boolean arrays
body: ```python
from sklearn.impute import SimpleImputer
import pandas as pd
X2 = pd.DataFrame({'a': [True, False, True, False, None]})
SimpleImputer(strategy='most_frequent').fit_transform(X2)
```

> TypeError: '<' not supported between instances of 'NoneType' and 'bool'

Interestingly this works when the remaining types are float, or when using ``np.NaN`` for the missing value.

Potentially related to #17625, where I suggested treating ``None`` as ``np.NaN``. 

FYI ``X2.a.unique()`` gives the expected result, ``np.unique(X2.a)`` errors. I'm not sure if using the  ``__array_function__`` protocol might help us here, we might be casting to numpy arrays before hitting the unique.
user: amueller
issue_18454: state: open
title: Feature request: regression report
body: #### Describe the workflow you want to enable

I'd like an alternative to the wildely used [sklearn.metrics.classification_report](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html) for regression problems

#### Describe your proposed solution
I purpose to implement it as `sklearn.metrics.regression_report`,  following code is a draft I code my self to re-use:

```python
from sklearn.metrics import mean_absolute_error, mean_squared_error, max_error, median_absolute_error, r2_score, explained_variance_score
import numpy as np

def regression_report(y_true, y_pred):
    
    error = y_true - y_pred
    percentil = [5,25,50,75,95]
    percentil_value = np.percentile(error, percentil)
    
    metrics = [
        ('mean absolute error', mean_absolute_error(y_true, y_pred)),
        ('median absolute error', median_absolute_error(y_true, y_pred)),
        ('mean squared error', mean_squared_error(y_true, y_pred)),
        ('max error', max_error(y_true, y_pred)),
        ('r2 score', r2_score(y_true, y_pred)),
        ('explained variance score', explained_variance_score(y_true, y_pred))
    ]
    
    print('Metrics for regression:')
    for metric_name, metric_value in metrics:
        print(f'{metric_name:>25s}: {metric_value: >20.3f}')
        
    print('\nPercentiles:')
    for p, pv in zip(percentil, percentil_value):
        print(f'{p: 25d}: {pv:>20.3f}')
```

Usage example:
```python
>>> from sklearn.metrics import regression_report
>>> regression_report(y_train, y_train_pred)
>>> Metrics for regression:
      mean absolute error:               10.176
    median absolute error:                6.714
       mean squared error:              247.877
                max error:               82.595
                 r2 score:                0.565
 explained variance score:                0.574

Percentiles:
                        5:              -14.780
                       25:               -6.320
                       50:               -0.271
                       75:                7.300
                       95:               34.608
```
#### Describe alternatives you've considered, if relevant
Relevant metrics and inpout/output format should be discussed by the community according to its interest and usage, and keeping the interfaces as similar as possible to the related `classification_report`.

#### Additional context

I'm able to implement and raise a PR if there is interest on it. I'd love to contribute to this library, Just would like to know if there is interest for this and how/which metrics to include in the report



user: rola93
issue_18448: state: open
title: Minimize memory usage in Tree based models by allowing for change of internal dtype
body: Internally in the `Tree` class, a float64/double type is used for the `value` array (https://github.com/scikit-learn/scikit-learn/blob/0.23.2/sklearn/tree/_tree.pyx#L547). This seems excessive in many cases. If it was possible to use other dtypes for such cases, a significant reduction in required memory would be obtainable.

A motivating example:
```python
X, y = make_classification(n_samples=int(1e5), n_informative=10, n_features=150, n_classes=450, random_state=8811)
simple_model = RandomForestClassifier(n_estimators=1)
simple_model = simple_model.fit(X, y)

print(simple_model.estimators_[0].tree_.value.nbytes / 1024**2)   # This is 98% of the memory required for the entire model
366.8231964111328

print(np.unique(simple_model.estimators_[0].tree_.value))  # Does not really need to be float64. Could be e.g. uint16.
array([  0.,   1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,
        11.,  12.,  13.,  14.,  15.,  16.,  17.,  18.,  19.,  20.,  21.,
        22.,  23.,  24.,  25.,  26.,  27.,  28.,  29.,  30.,  31.,  32.,
        33.,  34.,  35.,  36.,  37.,  38.,  39.,  40.,  41.,  42.,  43.,
        44.,  45.,  46.,  47.,  48.,  49.,  50.,  51.,  52.,  53.,  54.,
        55.,  56.,  57.,  58.,  59.,  60.,  61.,  62.,  63.,  64.,  65.,
        66.,  67.,  68.,  69.,  70.,  71.,  72.,  73.,  74.,  75.,  76.,
        77.,  78.,  79.,  80.,  81.,  82.,  83.,  84.,  85.,  86.,  87.,
        88.,  89.,  90.,  91.,  92.,  93.,  94.,  95.,  96.,  97.,  98.,
        99., 100., 101., 102., 103., 104., 105., 106., 107., 108., 109.,
       110., 111., 112., 113., 114., 115., 116., 117., 118., 119., 120.,
       121., 122., 123., 124., 125., 126., 127., 128., 129., 130., 131.,
       132., 133., 134., 135., 136., 137., 138., 139., 140., 141., 142.,
       143., 144., 145., 146., 147., 148., 149., 150., 151., 152., 153.,
       154., 155., 156., 157., 158., 159., 160., 161., 162., 163., 164.,
       165., 166., 167., 168., 169., 170., 171., 172., 173., 174., 175.,
       176., 177., 178., 179., 180., 182., 183., 184., 185., 187., 188.,
       189., 190., 191., 192., 193., 194., 195., 196., 197., 198., 199.,
       200., 201., 202., 203., 204., 205., 206., 207., 208., 209., 210.,
       211., 212., 213., 214., 215., 216., 217., 218., 219., 220., 221.,
       222., 223., 224., 225., 226., 227., 228., 229., 230., 231., 232.,
       233., 234., 235., 236., 237., 238., 239., 240., 241., 242., 243.,
       244., 245., 247., 248., 249., 250., 251., 252., 253., 254., 255.,
       256., 257., 267., 268., 270.])
```
In the above example, the memory requirements may be lowered by a factor of ~4 by switching to uint16. If there were fewer classes in the model, one could even have settled for an uint8. For a regression model this probably needs to be a float32. However, even for the classification case, using a float32 would allow for a memory reduction by a factor of ~2. As a side note, since the input data `X` is cast to float32 anyway, it does not seem unreasonable to have float32 be the default.

In my real world case, I have a RandomForestClassifier which takes up about ~10 GiB RAM. If I could obtain a memory reduction by a factor of ~4, I would only need a ~4 GiB VM when hosting and/or training this model in the cloud - which would be a significant saving for me.

Thus, ideally, I would like to have classes like `RandomForestClassifier` or `RandomForestRegressor` intelligently pick the dtype of the `Tree` `value` array - or at least let me specify it myself. 

Any feedback on the feasibility of this suggestion is highly appreciated. My Cython skills are a little inadequate to fully understand the internals of the `Tree` class.

This may be related to https://github.com/scikit-learn/scikit-learn/issues/14747 and https://github.com/scikit-learn/scikit-learn/issues/11165.
user: Chroxvi
issue_18443: state: open
title: MinMaxScaler output datatype
body: #### Describe the workflow you want to enable
Currently, applying a MinMaxScaler to data that includes features with small datatypes like int8 results in float64 output. I would like to have a way to output to a datatype that is also some low precision type such as float16, and I don't believe this is supported in the MinMaxScaler today without applying another transformation after applying the minmax scaler. This likely applies to other scaling functions as well.

I would like this capability in order to avoid running out of memory on large-ish datasets that could be operated on in one VM, but can't after many of my columns turn into higher-than-necessary-precision datatypes.

#### Describe your proposed solution
I'd modify MinMaxScaler to accept an optional output data type argument and then cast values to that type while performing the necessary arithmetic for scaling. 

#### Describe alternatives you've considered, if relevant
The casting operation could happen at many points, including after performing arithmetic in a higher precision type like float64, which could be important in some use cases to avoid loss of precision.

#### Additional context
Here's an example of a case based on MinMaxScaler docs where applying the MinMaxScaler takes int8 data and turns the result into float64
```
import numpy as np
from sklearn.preprocessing import MinMaxScaler

data = np.array([[-1, 2], [-1, 6], [0, 10], [1, 18]], dtype=np.int8)
scaler = MinMaxScaler()

assert(data.dtype==np.int8)
assert(scaler.fit_transform(data).dtype == np.float64)
```

user: nkelly13
labels: Needs Decision: Requires decision
issue_18422: state: open
title: [MRG] FEA Add MDLP discretization
body: #### Reference Issues/PRs

Related with #15551.

#### What does this implement/fix? Explain your changes.

This PR adds the [MDLP](https://www.ijcai.org/Proceedings/93-2/Papers/022.pdf) supervised discretization (~3892 cites) to the `sklearn.preprocessing` module.

#### Any other comments?

The code is (partially) based on the decision tree one, but caching results for optimization.

#### TODO

- [x] Test suite
- [x] User Guide
- [x] API Reference
- [x] Examples
user: alfaro96
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_18413: state: open
title: [FEAT] Weighting option in MDS
body: Fixes https://github.com/scikit-learn/scikit-learn/issues/6828

#### What does this implement/fix? Explain your changes.

This PR introduced an option for MDS to incorporate a weight matrix (or importance matrix) during its nonmetric optimization process inside SMACOF. In short, instead of considering all precomputed pairwise distances, there is a weight assigned to each distance. By default, the weight matrix will be all ones, meaning that the optimization objective treating all distances the same weight.

For publication references, the weighted version of nonmetric MDS is introduced in page 10 of (de Leeuw & Mair, 2008). 

This function is especially useful in illuminating blockwise structures in data given different meta-data. For instance, in neuroscience, a weighted MDS can enlighten a more structural representational space given the priors of different stimulus category within the data.   

#### Any other comments?

user: doerlbh
issue_18403: state: open
title: Upgrade DictVectorizer to use 64 bit indices instead of 32 bit
body: #### Describe the workflow you want to enable
Currently, in DictVectorizer, the _transform function is limited to 32 bit indices, meaning that there is a limit of ~2B rows/cols in the resultant matrix. 

This issue seeks to enable using DictVectorizer on larger datasets by increasing this to work with 64 bit values.

#### Describe your proposed solution
Update the dtypes for the indices in [_dict_vectorizer.py](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/_dict_vectorizer.py) as follows:
* from `np.int32` to `np.int64`
* from `np.intc` to `np.int_`
* update from `array("i")` to `array("l")` to get the 64 bit signed long instead of signed int


user: E-Aho
issue_18390: state: open
title: [MRG] Added option to return raw predictions from cross_validate
body: Fix for #13478
Created new PR #15907 due to merge conflict.

This allows the user to extract predictions or prediction probabilities using 'return_predictions' argument for cross_validate. The argument takes 'predict_proba' or 'predict' as a string to return the predictions.

The approach has been to have `_fit_and_score` run inference on the cross-validation split and return it with the indices. Once all the indices and predictions have been returned to the `cross_validate` function, they are ordered according to the dataset and returned.

```
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_validate
from sklearn.datasets._samples_generator import make_classification

n_samples = 25
n_classes = 2
clf = LogisticRegression(random_state=0)
X, y = make_classification(n_samples=n_samples, n_classes=n_classes)

ret = cross_validate(clf, X, y, return_train_score=False,
                     return_estimator=True, return_predictions='predict')
ret['predictions']
>>> array([0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1,
       0, 1, 1])
```

@cmarmo could you take a look please?
user: okz12
issue_18350: state: open
title: Negative label value in silhouette_score function
body: An additional point is needed in the documentation for the **silhouette coefficient score** (Function: _sklearn.metrics.silhouette_score_, Documentation Page: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html).

One of the input parameters to the silhouette coefficient is the clustering labels (second parameter "labels"). In the output of the **DBSCAN** algorithm (https://scikit-learn.org/stable/modules/generated/sklearn.cluster.dbscan.html), the label -1 is assigned to the noise points (noise points are not part of a cluster). A programmer expects that when an input label array (for example, -1, 0, 0, 1, 1, 1) is fed into the _silhouette_score_ method, the label -1 is skipped. However, the label -1 is considered as a cluster in the silhouette coefficient calculation.

The following line can be added in the documentation for the function _silhouette_score_.

"Clustering label should start from index 0."

In the above case, a programmer will remove -1 from the label array manually.
user: macgok
issue_18309: state: open
title: Add transformation option to inspect functions
body: #### Describe the workflow you want to enable

I like the inspect module very much. Sometimes, interpretation of a model is more natural on a different scale (e.g. log) than on the scale of the predictions.

Here some examples:

1. We fit a GLM. It is often more natural to inspect the model on the linear scale of its link function, not on the scale of the response variable.
2. We fit a Poisson regression in XGBoost/LGB. These models are fitted on log scale, so it might be worth inspecting them on the log scale, even if it is only to compare with a benchmark GLM model.

#### Describe your proposed solution

Add an argument `transformation` to all explainers (partial dependence, permutation importance). By default, it is None (or the identity). The user can pass e.g. `np.log` to allow evaluation on the log scale. 

- Partial dependence plot: Here, it suffices to transform the predictions before averaging them.

- Permutation importance: Here, both the response and the predictions need to be transformed. The scorer must be in line with the transformation and provided by the user.

#### Describe alternatives you've considered, if relevant

An alternative would be to change the prediction function of the Classifier/Regressor.


user: mayer79
labels: Needs Decision: Requires decision
issue_18307: state: open
title: RFC: remove support of pos_label inference with str in brier_scorer_label
body: Currently, the following list of metrics will raise an error if `str` are provided as labels and that `pos_label` is not defined and needed to compute the metrics:

* `average_precision_score`
* `f1_score`
* `fbeta_score`
* `jaccard_score`
* `precision_recall_curve`
* `precision_score`
* `recall_score`
* `roc_curve`

We try to make sure that the error message is consistent in https://github.com/scikit-learn/scikit-learn/pull/18192

`brier_score_loss` should supposedly follow the behaviour than the other metrics. However, it seems that up-to-now, it does some inference when labels are string.

Thus, 2 questions can be raised:

* Is it normal that `brier_score_loss` can do such inference or should it raise a similar error as other metrics?
* If it should raise an error, shall we open consider as a bug fix or make a deprecation cycle?
user: glemaitre
issue_18304: state: open
title: partial_fit for StackingClassifier/Regressor?
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

This is mainly a question of whether such a feature would make sense. I am a beginner data scientist (though I have some background in software engineering), so please excuse any misunderstandings on my part.

It is my understanding that the StackingClassifier/Regressor takes a list of estimators, fits them onto the provided data, and uses their outputs as data for the final estimator. When `fit` is called, all the estimators are fitted. Taking that into account, would a `partial_fit` method make sense here? I envision each call to `partial_fit` fitting one estimator, appending its output to an array, and then fitting the final estimator on the current version of the array. Each call to `partial_fit` would increase the size of the array, and after `n` calls of `partial_fit`, where `n` is the number of estimators, the result would be identical to the result of `fit`.

Would such a feature make sense? I imagine it being useful for tuning of the final estimator with algorithms that support early stopping using `partial_fit`, as fitting a large StackingClassifier/Regressor can take a long time.

I would be happy to try and implement such a feature if it indeed is a good idea.

user: Yard1
issue_18292: state: open
title: [MRG] Test __array_function__ not called in non-estimator API  (#15865)
body: In NEP18 numpy created a dispatch mechanism that enables data types to override the implementation of numpy functions.
The principle is that when a numpy function receives input parameters that implement \_\_array_function\_\_ it will delegate the execution to the \_\_array_function\_\_ of this parameters.
So if we pass to a numpy function parameters of type dask.array (that implements \_\_array_function\_\_) the result will not be a numpy.ndarray as before but a dask.array instead.  This can cause troubles to the scikit-learn code.
in PR #14702 it was made sure that all scikit-learn estimators are protected from such a scenario.
In this PR we will ensure that other functions in scikit-learn like `cross_validate` or `permutation_importance` are also protected.
user: alexshacked
labels: module:test-suite: everything related to our tests
issue_18289: state: open
title: Support multioutput in VotingClassifier and VotingRegressor
body: #### Describe the workflow you want to enable
Instead of having to manually take the mean of an ensemble of multioutput estimators (as done in https://scikit-learn.org/stable/auto_examples/multioutput/plot_classifier_chain_yeast.html), it would be nice to support multioutput algorithms in `VotingClassifier`/`VotingRegressor`. This could perform the same weighted voting to each output.

In the https://scikit-learn.org/stable/auto_examples/multioutput/plot_classifier_chain_yeast.html example, this would replace:
```
# Fit an ensemble of logistic regression classifier chains and take the
# take the average prediction of all the chains.
chains = [ClassifierChain(base_lr, order='random', random_state=i)
          for i in range(10)]
for chain in chains:
    chain.fit(X_train, Y_train)

Y_pred_chains = np.array([chain.predict(X_test) for chain in
                          chains])
Y_pred_ensemble = Y_pred_chains.mean(axis=0)
```
with 
```
# Fit an ensemble of logistic regression classifier chains and take the
# take the average prediction of all the chains.
ensemble_clf = VotingClassifier([ClassifierChain(base_lr, order='random', random_state=i) for i in range(10)])
ensemble_clf.fit(X_train, Y_train)
Y_pred_ensemble = ensemble_clf.predict(X_test)
```
user: charlesbmi
issue_18268: state: open
title: Feature Request: function to calculate Expected Calibration Error (ECE)
body: #### Describe the workflow you want to enable

I would like to add the ability to calculate Expected Calibration Error (ECE) within scikit-learn. ECE is defined in equation (3) from [Guo et al. *On Calibration of Modern Neural Networks.* (2017).](https://arxiv.org/abs/1706.04599) This is a well-cited paper (over 700 citations as of Aug. 26, 2020), and the ECE metric is now widely used in academic papers on model calibration. There is even a method now in TensorFlow Probability for computing ECE: [tfp.stats.expected_calibration_error()](https://www.tensorflow.org/probability/api_docs/python/tfp/stats/expected_calibration_error).


#### Describe your proposed solution

I can see several possibilities for how the ECE calculation could be added to scikit-learn, with pros and cons for each:

1. make ECE an extra return value of `sklearn.calibration.calibration_curve()`. This method currently already does 99% of the work towards calculating ECE. Computing ECE is simply a one-liner added to the end of that method:

    ```python
    def calibration_curve(...):
        ...
        ece = np.sum(np.abs(prob_true - prob_pred) * (bin_total[nonzero] / len(y_true)))
        return prob_true, prob_pred, ece
    ```

    The downside is that introducing a 3rd return value is a breaking change. One possible mitigation is to add a boolean `return_ece` parameter to the function definition with a default value of `False`, and only return `ece` if `return_ece=True`.

2. add a dedicated function to `sklearn.metrics`, i.e., `sklearn.metrics.expected_calibration_error(y_true, y_pred)`. Downside here is that it would need to re-compute everything that calibration_curve() already computes, so there is a performance penalty for a user who wants to both calculate ECE and plot a calibration curve.

3. add a dedicated function to `sklearn.calibration`, i.e., `sklearn.calibration.expected_calibration_error(y_true, y_pred)`. This would keep the ECE calibration within the calibration subpackage. Same downside as in option 2.

#### Additional context

I am happy to write the code and tests to add ECE calculation to scikit-learn. After all, the code to calculate ECE is just a one line addition to `calibration_curve()`. However, as I am not a regular contributor to scikit-learn, I am unfamiliar with the best place within the scikit-learn library to add such a feature. Please advise!
user: chrisyeh96
issue_18257: state: open
title: RFC Plan for reworking the tutorials
body: On top of the Examples and the User Guide, we have a [Tutorial](https://scikit-learn.org/stable/tutorial/index.html#) section. This part of the docs is quite old, and some of it became redundant with the UG and / or the examples. Lots of users will be drawn to the tutorials page at some point, so I feel like it's important that we give it the attention it deserves, just as we do for the UG.

Here's a proposal on how to deal with each section of the current Tutorial page:

1. *An introduction to machine learning with scikit-learn*: This is what used to be our "Quick Start" guide before it was re-written in https://github.com/scikit-learn/scikit-learn/pull/14920. I find this doc quite strange and unintuitive, because it deals with very different levels of expertise: it starts by introducing ML, and it ends up by detailing how to cast data to `flaot64`.

	I would suggest to completely remove this guide, or alternatively, to somewhat merge it with the next one (2.)

2. *A tutorial on statistical-learning for scientific data processing*: This looks like excellent material (from @GaelVaroquaux I think?), and I believe this should actually be our whole "Tutorials" page. There are however a few issues with this guide:
	- it's quite old (up to 9 years old), so some sections might not follow our current best practice
	- ~there are layout issues~ as e.g. [here](https://scikit-learn.org/stable/tutorial/statistical_inference/supervised_learning.html#nearest-neighbor-and-the-curse-of-dimensionality) (fixed in #18261)
	- it's hard to navigate between sections, since the sidebar isn't updated as one might expect.

	I would suggest to:
	- make a careful pass over it and update it to our recommended best practices / introduce recent tools
	- ~fix the layout issues~ (done in #18261)
	- fix the sidebar issues
	- make this the entire "Tutorials" section of the docs
	- maybe integrate stuff from 1.

3. *Working With Text Data*: This thing looks like it could be just an Example, so we can probably remove it from the Tutorials page. I haven't checked but I  think we already have similar examples, so we could merge this one with some others

4. *Choosing the right estimator*: I think our users love this pic, and @amueller (the author) now hates it ^^. Maybe it could just stay there, or we could transform it into an example.

5. *External Resources, Videos and Talks*: this section presents some super old tutorial videos, the most recent is from 2013. I think we could:
	- update it to the more recent tutorials e.g. those given recently at Scipy and EuroScipy.
	- Make it a subsection of 2., if 2. becomes the whole "Tutorials" page as suggested.


Of course these are only suggestions, I'm happy to discuss alternatives.
user: NicolasHug
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_18223: state: open
title: Add Altmann's permutation importance (PIMP) and Janitza's New Testing Approach (NTA)
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

I think it would be nice to implement Altmann's permutation importance (or PIMP) in scikit - original paper is https://academic.oup.com/bioinformatics/article/26/10/1340/193348. The idea was already proposed in https://github.com/scikit-learn/scikit-learn/issues/11187#issuecomment-567442138 but I think nothing came of it. 

The method is already implemented in 2 or 3 packages for R that I know of, one being the original release by the authors and the other two being the Vita package by Janitza et al. and the other being Ranger. However, I believe it is being hampered by the very slow implementation of random forests (in my experience) that it leverages in R (both randomForest and ranger), and it could benefit and see greater use if it took advantage of scikit's RF implementation (by my experiments, using the same dataset gives a training time of ~70 seconds in scikit and ~30 minutes in ranger, likely double in randomForest).

The benefits are that it is easier/faster to implement than the conditional permutation scheme by Strobl et al. while leaving the dependence between features untouched, and that for a large number of features it would be faster to compute than standard permutation importance (altough PIMP requires retraining the model for each permutation, classical permutation importance does not). 
As for the drawbacks of this method, I believe the main one would be the number of permutations needed to get reliable p-values in the end. The original paper suggests between 50-100 permutations of the response vector, however one of the implementations I found (I believe in ranger in R) suggests that >>100 are needed for accurate p-values.
Also, it is still based on Random Forests and Gini Impurity measures, so I believe there is still a bias towards high cardinality features and correlated features, but the whole distribution estimation/p-value procedure should help filter out some redundant or otherwise statistically insignificant importance scores.

On the note of speed I should also mention the paper by Janitza et. al (https://link.springer.com/article/10.1007/s11634-016-0276-4) which proposes an algorithm (NTA) that is 100 to 125 times faster than Altmann's PIMP, altough it needs a high number of features in the dataset and for many of them to have non-positive importance scores. I believe this method doesn't respect scikit's guidelines for the inclusion of a new algorithm given the number of citations, but it could still be useful to implement; furthermore, Degenhardt et al. indicate it as the best feature selection method along with Boruta in their "Evaluation of variable selection methods for random forests and omics data sets" (https://europepmc.org/article/med/29045534)
user: Devilmoon
issue_18218: state: open
title: Make passthrough argument take a list of features
body: https://github.com/scikit-learn/scikit-learn/blob/0fb307bf39bbdacd6ed713c00724f8f871d60370/sklearn/ensemble/_stacking.py#L87-L90

In this code, it would be helpful to allow user to select a subset of features to include in the final estimator instead of just having a bool state whether to append the full feature set. As a simple start, could just take a list column indices to include in the passthrough set.

It would also be nice to accept an ndarry or dataframe of features that could just be appended to the final estimator. For example, I have a desire to add some features to the dataset only after the preliminary estimator predictions have been made.

user: jvschoen
issue_18205: state: open
title: max_iter limit argument inconsistencies between solvers.
body: This question likely stems from my poor understanding of sklearn and apologies if this has been answered but I was wondering why for some estimators, such as `SVC()`, the `max_iter` argument can be set to -1 for removing the limit but for other estimators such as `SGDClassifier()` there needs to be a hard limit, otherwise there is the error `ValueError: max_iter must be > zero. Got -1.000000`.


user: browshanravan
issue_18187: state: open
title: feature_importance causes a BSOD on Windows 10
body: 
#### Describe the bug
Running permutation_importance on a medium-sized data set results in a BSOD on Windows 10. The dataset is 470605 x 332, code is running in a Jupyter notebook, Python version 3.7.6, scikit version 0.22.1.
The BSOD is a KERNEL_SECURITY_CHECK_FAILURE, with ERROR_CODE: `(NTSTATUS) 0xc0000409 - The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application.`
The machine has a Ryzen 5 3600 with 16GB of RAM.

#### Steps/Code to Reproduce

```python
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
rf = RandomForestClassifier(n_estimators = 250,
                           n_jobs = -1,
                           oob_score = True,
                           bootstrap = True,
                           random_state = 42)
rf.fit(X_train, y_train)
permImp = permutation_importance(rf,
                                 X_val,
                                 y_val,
                                 scoring='f1',
                                 n_repeats=5,
                                 n_jobs=-1,
                                 random_state=42)
```

#### Expected Results
No BSOD, permutation importance computed.

#### Actual Results
BSOD after ~1-2 minutes

#### Versions
> sklearn.show_versions()

System:
    python: 3.7.6 (default, Jan  8 2020, 20:23:39) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\lucag\anaconda3\python.exe
   machine: Windows-10-10.0.18362-SP0

Python dependencies:
       pip: 20.0.2
setuptools: 45.2.0.post20200210
   sklearn: 0.22.1
     numpy: 1.18.1
     scipy: 1.4.1
    Cython: 0.29.15
    pandas: 1.0.1
matplotlib: 3.1.3
    joblib: 0.14.1

Built with OpenMP: True
<!-- Thanks for contributing! -->

user: Devilmoon
issue_18186: state: open
title: Possible to parallelize sklearn.impute.KNNImputer?
body: Greetings!

Do you think it might be possible to parallelize the algorithm for `sklearn.impute.KNNImputer` in the future?

scikit-learn's implementation of `sklearn.neighbors.KNeighborsClassifier` accepts an `n_jobs` parameter to achieve this, but the corresponding imputation function does not and can be quite slow for large datasets.


user: khughitt
issue_18177: state: open
title: Spectral Embedding with Asymmetric Matrices/ Directed Graphs
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
Currently, the `sklearn.manifold.SpectralEmbedding` is restricted to symmetric affinity matrices, and if an asymmetric matrix is passed it is converted through `sklearn.utils.validation.check_symmetric` into a symmetric matrix. However, in doing so, one loses the underlying asymmetries and potential directional clusters present in the adjacency matrix of the directed graph input.
#### Describe your proposed solution
The algorithms I propose adding use singular value decomposition, as opposed to eignendecomposition, and a modified Laplacian to perform spectral embedding on directed graphs/asymmetric matrices.   Specifically I would like to propose adding [Adjacency](https://www.tandfonline.com/doi/full/10.1080/01621459.2012.699795) / [Laplacian](https://www.pnas.org/content/pnas/113/45/12679.full.pdf) spectral embedding, ASE and LSE respectively.  My thoughts would be to add a new class,`sklearn.manifold.DirectedSpectralEmbedding`, which users may call directly, or is dispatched to when an asymmetric matrix is passed to  `sklearn.manifold.SpectralEmbedding`. Similarly to `SpectralEmbedding`, users would specify between ASE and LSE through an `affinity` parameter.


#### Additional context
These algorithms have been implemented in [GraSPy,](https://graspy.neurodata.io/tutorials/embedding/adjacencyspectralembed) (available at [ASE](https://github.com/neurodata/graspy/blob/ase-embed/graspy/embed/ase.py) and [LSE)](https://github.com/neurodata/graspy/blob/ase-embed/graspy/embed/lse.py), inputing a graph represented as a dense or sparse matrix, and returning the appropriate embedding. 

user: asaadeldin11
issue_18155: state: open
title: [MRG] Fix Single linkage option in Agglomerative causes MemoryError for very large numbers
body: #### Reference Issues/PRs
Fixes #17960


#### What does this implement/fix? Explain your changes.
Issue #17960 (Single linkage option in Agglomerative causes MemoryError for very large numbers) causes the hierarchical tree to become highly connected. As a consequence, as we are trying to explore the hierarchical tree, we keep re-exploring the same nodes. I implemented a simple fix by keeping track of what nodes we have already visited during our exploration and that fixes the issue above.
user: juliusvering
issue_18141: state: open
title: ENH allow to pass str or scorer to make_scorer
body: This PR proposes to add the option to pass a scoring function name or a scorer in order to build a scorer via `make_scorer`.
In some way, it would simplify the scorer API. For instance, the following example is raising an error because the class 9 is not present in the fold during cross-validation:

```python
import sklearn.datasets
import sklearn.linear_model
import numpy as np

X, y = sklearn.datasets.load_digits(return_X_y=True)

n9 = 2

y9 = y[y == 9]
X9 = X[y == 9]

X = X[y != 9]
y = y[y != 9]

X = np.vstack([X, X9[0:n9]])
y = np.hstack([y, y9[0:n9]])

multi_class = "multinomial"
scoring = "neg_log_loss"
random_state = 1

model = sklearn.linear_model.LogisticRegressionCV(
    multi_class=multi_class, scoring=scoring, random_state=random_state
)
model.fit(X, y)
```

The way to solve it is to use a scorer. However, one needs to know what all the `make_scorer` parameters mean:

```python
    multi_class = "multinomial"
    labels = np.unique(y)
    scoring = make_scorer(
        log_loss, greater_is_better=False, labels=labels, needs_proba=True
    )
    random_state = 1

    model = LogisticRegressionCV(
        multi_class=multi_class, scoring=scoring, random_state=random_state
    )
    model.fit(X, y)
```

I am thinking that the following would be simpler where some parameters can be inferred from the base scorer.


```python
    multi_class = "multinomial"
    labels = np.unique(y)
    scoring = make_scorer("neg_log_loss", labels=labels)
    random_state = 1

    model = LogisticRegressionCV(
        multi_class=multi_class, scoring=scoring, random_state=random_state
    )
    model.fit(X, y)
```
user: glemaitre
issue_18130: state: open
title: A small fix of calibration_curve implementation
body: https://github.com/scikit-learn/scikit-learn/blob/0fb307bf39bbdacd6ed713c00724f8f871d60370/sklearn/calibration.py#L592-L594

I suggest the argument `minlength` of `np.bincount`  be changed from `len(bins)` to `n_bins`. Although the former won't produce a bug since the extra bin has no samples and will be excluded, the latter seems better because it doesn't create an extra bin and is easier to understand.
user: szywise
issue_18122: state: open
title: cosine_similarity function produces results greater than 1.0 
body: https://github.com/scikit-learn/scikit-learn/blob/0fb307bf39bbdacd6ed713c00724f8f871d60370/sklearn/metrics/pairwise.py#L1144
reproduce the greater than 1.0 case:
```python
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

a = np.array([[ 1,  2],
       [ 1,  4],
       [ 1,  0],
       [10,  2],
       [10,  4],
       [10,  0]])
b = np.array([[10.,  2.],
       [ 1.,  2.]])
cosine_similarity(a, b)[3][0]
1.0000000000000002
```
centos-7.2
python: 3.6.10
sklearn: 0.23.2
numpy: 1.19.1
scipy: 1.5.2

may be we should use K =  safe_sparse_dot(X, Y) / sqrt(||X||^2 * ||Y||^2) instead of normarlize X,Y first
user: wwz58
labels: Needs Decision: Requires decision
issue_18118: state: open
title: Rewrite fast_mcd to be more memory-efficient
body: Fixes #18096

1. Instead of storing intermediate results in a large array that may not fit in memory, this processes intermediate results as soon as they are available. In addition, the different cases (`n_samples <= 500`; `500 < n_samples <= 1500`; `1500 < n_samples`) are separated in a way that is hopefully more transparent.

I've done my best to not change the logic, and I seem to obtain identical results with datasets of varying sizes, and similar run times.

PS: I cannot see in the paper why `n_trials = 30` if `n_samples <= 500`, but I've left this alone for now.

2. When `support_fraction == 1`, the FAST-MCD paper recommends calculating covariance directly, and this is much more efficient (e.g. no need to run 10–500 trials if they all produce the same result), so I've changed `fit` to call `super().fit` in that case. This way, it becomes possible to use `EllipticEnvelope` for novelty detection with `support_fraction = 1` and not suffer an unnecessary performance penalty.

user: oulenz
issue_18101: state: open
title: pos_label in plot_precision_recall_curve and plot_roc_curve
body: With the explanation regarding `pos_label` provided by @amueller in https://github.com/scikit-learn/scikit-learn/pull/17704#issuecomment-669481297 

> My understanding is that pos_label is not the semantics of the problem, but the semantics of the predict_proba/decision function that you passed. If you have some classes, 'a' and 'b' and a decision function, then pos_label='a' means that high values of the decision function are supposed to correspond to the class 'a' being likely.

I think we should either remove `pos_label` from `plot_precision_recall_curve` and `plot_roc_curve` or rename it to `positive_class`. Naming the parameter differently was originally suggested in https://github.com/scikit-learn/scikit-learn/issues/15573
user: thomasjpfan
issue_18096: state: open
title: `fast_mcd` results in `IndexError` with certain large datasets
body: https://github.com/scikit-learn/scikit-learn/blob/12e30fa10fb8dcbbed642cae3bd5a2b25d1ba11b/sklearn/covariance/_robust_covariance.py#L435

`fast_mcd` tries to create an array of shape `(n/30,  m, m)` (where `n` and `m` are the number of instances and features). If that  causes a `MemoryError`, it catches it and instead creates an array of shape `(10, m, m)`. However, it then still tries to index this array in the first dimension with indices up to `n/150`. Which leads to an IndexError if `n > 1500`.

(I may have calculated some of these numbers wrong, but the essence should be correct. Curiously, the code that follows branches depending on whether `n > 1500`, but this seems to be a coincidence? It might make sense to also branch on `n > 1500` here.)

At the least, I think the `IndexError` should be caught, in which case the user should be informed that their dataset is too large. But I also think the code should try harder, by possibly creating a larger array (yet not so large as to cause a `MemoryError`), or alternatively further reducing the index size.

One approach would be to enforce `n_best_tot = n_subsets * n_best_sub` and determine a maximal `n_best_sub` between 1 and 10, as well as to reduce the size of `n_subsets` if necessary.

(I encountered this while running `EllipticEnvelope` on a high-dimensional dataset for benchmark purposes.)
user: oulenz
issue_18083: state: open
title: SpectralClustering with assign_labels='discretize' returns less clusters than specified in n_clusters
body: I'm not sure if this is a bug or an undocumented feature (or at least I couldn't find any info about it).

Using sklearn version 0.22.1.

The following code is producing the behavior mentioned in the title:

```python
clustering = SpectralClustering(n_clusters=4, affinity='rbf', assign_labels='discretize', random_state=0).fit(X)
np.unique(clustering.labels_) shows just 3 labels. 
```

X is a np.array of shape (258, 257).
This happens only with some random seeds. I was not able to reproduce it with assign_labels='kmeans' (running multiple times).
Also the labels of the clusters are not consecutive as if less than n_clusters where picked out of the n_components.
user: andins
issue_18073: state: open
title: I think OneHotEncoder should be able to handle mixed data types
body: I think it's very practical to OneHotEncode columns that have mixed datatypes. Pandas (or some other dataframe reader) will unlikely read in the true datatype of every column correctly. Pandas reading in a column as an integer does not necessarily mean the column should be treated as an integer in sklearn.

Example: pandas will read in a column that contains codes of certain products (e.g. instrument_type will contain 10 integers in the dataset - 1000, 1001, ... 1009, NaN, and 'guitar'). There are only 12 categories. I cannot one hot encode this column unless I convert it to string (using .astype(str)). If there were no NaN this would still be a problem, but with NaN I can simple do a fillna('nan'), and then passed that to the final pipeline with FunctionTransformer. If only fitting something for exploratory purposes, the .astype(str) is a simple and quick solution. However, it is not efficient in a production pipeline. I found myself having to create a custom class that converts each column I want to OH encode to astype(str), so I can then implement the OH encoder in production even though pandas will interpret that column as an integer.

The solution for this would be to have an input parameter that allows OneHotEncoder (and OrdinalEncoder) to perform on columns with mixed datatypes. If the user does this when he/she shouldn't, that should be user error, but not decided by the sklearn package. What are the major reasons this hasn't bee implemented? Thoughts?

Code example
```
def na_impute(X, fill_value = 'nan', return_string = True):
        if return_string:
            return X.fillna(fill_value).astype(str)
        else:
            return X.fillna(fill_value)

oh_pipe = make_pipeline(
            FunctionTransformer(na_impute),
            OneHotEncoder(handle_unknown = 'ignore', sparse = False, drop = 'if_binary') # fails because drop = 'if_binary' and handle_unknown = 'ignore'
                           )
```

user: melgazar9
issue_18066: state: open
title: Moving a variation of _validate_estimators from fit into the stacking constructor
body: In stacking, the ```estimators``` parameter is a list of ```(str, estimator)```, although this behavior is not enforced until the ```fit``` method:

```python
from sklearn.datasets import load_boston
from sklearn.linear_model import Ridge
from sklearn.ensemble import StackingRegressor

X, y = load_boston(return_X_y=True)
reg = StackingRegressor([Ridge(), Ridge()])
```

With this example, one may want to run ```reg.get_params()```, which will result in ```TypeError: cannot convert dictionary update sequence element #0 to a sequence```. This seems complicated for a user to debug, so I propose moving a variation of 
https://github.com/scikit-learn/scikit-learn/blob/973ec69d50410ccee0382ea9c203802b099436f1/sklearn/ensemble/_stacking.py#L138-L139

into the constructor, similar to 

https://github.com/scikit-learn/scikit-learn/blob/fd237278e895b42abe8d8d09105cbb82dc2cbba7/sklearn/pipeline.py#L830

in the Pipeline constructor. 

user: a-wozniakowski
issue_18057: state: open
title: Unexpected behavior when passing multiple parameter sets to RandomizedSearchCV
body: #### Describe the bug

Here is part of the documentation for the `param_distributions` parameter of [RandomizedSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html):

> If a list of dicts is given, first a dict is sampled uniformly, and then a parameter is sampled using that dict as above.

My interpretation is that if I pass a list of two dictionaries, then at each iteration:

- First, one of the two dictionaries will be chosen at random
- Then, a set of parameters within that dictionary will be chosen at random

I have found that that is **not** the case. Instead, I have found that if one of the two dictionaries has many more possible parameter combinations, then the larger dictionary will usually be chosen at each iteration.

#### Steps/Code to Reproduce

```python
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV

X, y = load_iris(return_X_y=True)
rf = RandomForestClassifier()

# 30 possible combinations
params1 = {}
params1['n_estimators'] = [10, 20, 30, 40, 50]
params1['min_samples_leaf'] = [1, 2, 3, 4, 5, 6]

# 120 possible combinations
params2 = {}
params2['n_estimators'] = [60, 70, 80]
params2['min_samples_leaf'] = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
params2['max_features'] = ['auto', None]
params2['bootstrap'] = [True, False]

params_both = [params1, params2]

rand = RandomizedSearchCV(rf, params_both, cv=5, scoring='accuracy', n_iter=50, random_state=1)
rand.fit(X, y)
print(sorted(rand.cv_results_['param_n_estimators']))
```

#### Expected Results

Since `n_iter=50`, I would expect that `params1` and `params2` would each be chosen about 25 times.

#### Actual Results

Here is the actual output of the last line:

```
[10, 20, 30, 40, 40, 50, 50, 50, 50, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 70, 70, 70, 70, 70, 70, 70, 70, 70, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80]
```

As you can see, `params1` (which had values 10 through 50) was chosen only 9 times, and whereas `params2` (which had values 60 through 80) was chosen 41 times.

#### Comments

There seem to be two possibilities:

1. The current behavior of `RandomizedSearchCV` **is** the desired behavior. In that case, I would propose tweaking the documentation to make this behavior more clear.
2. The current behavior of `RandomizedSearchCV` **is not** the desired behavior. In that case, I would propose fixing the behavior so that it matches the documentation.

I don't have a strong feeling about which behavior is the "optimal" behavior.

This was implemented in #14549 by @amueller, so he may have some insight on this!

#### Versions

```
System:
    python: 3.8.2 | packaged by conda-forge | (default, Apr 24 2020, 07:56:27)  [Clang 9.0.1 ]
executable: /Users/kevin/miniconda3/envs/sk23/bin/python
   machine: macOS-10.14.6-x86_64-i386-64bit

Python dependencies:
          pip: 20.1.1
   setuptools: 46.4.0.post20200518
      sklearn: 0.23.1
        numpy: 1.18.4
        scipy: 1.4.1
       Cython: None
       pandas: 1.0.3
   matplotlib: 3.2.1
       joblib: 0.15.1
threadpoolctl: 2.0.0

Built with OpenMP: True
```

user: justmarkham
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_18028: state: open
title: Support fit_params in stacking
body: Currently, there is no support for ```**fit_params``` in the ```fit``` method of ```_BaseStacking```:
https://github.com/scikit-learn/scikit-learn/blob/fd237278e895b42abe8d8d09105cbb82dc2cbba7/sklearn/ensemble/_stacking.py#L110

As introduced in issue [#15953](https://github.com/scikit-learn/scikit-learn/issues/15953) for ```_MultiOutputEstimator```, it seems natural to extend the utility to stacking. A proposed implementation in the base stacking class is as follows:

```python
from ..utils.validation import _check_fit_params

def fit(self, X, y, sample_weight=None, **fit_params):
    # Right before predictions = Parallel...
    if fit_params:
        fit_params = _check_fit_params(X, fit_params)
    else:
        fit_params = (dict(sample_weight=sample_weight)
                      if sample_weight is not None
                      else None)
    # Then, utilize fit_params in the parallelized cross_val_predict
```

Subsequently, alter the fit methods for ```StackingClassifier``` and ```StackingRegressor``` such that they support ```**fit_params```. If this is favorable, then I can write an implementation and start the pull request.

user: a-wozniakowski
issue_18027: state: open
title: Overflow error following standard scalar in LinearSVC: value to large to convert to npy_int32
body: I have a large matrix (150000 by 32000) I am trying to run a linear SVC using scikit-learn to perform a lasso regression 

``` lsvc = LinearSVC(C=0.01, penalty="l1", dual=False).fit(df, y) ```

I am receiving the following error text:

``` Traceback (most recent call last):
  File "scripts/Lasso_Feature_Selection.py", line 42, in <module>
    lsvc = LinearSVC(C=0.01, penalty="l1", dual=False).fit(df, y)
  File "/Strong/home/glickmanc/.conda/envs/py36/lib/python3.6/site-packages/sklearn/svm/_classes.py", line 243, in fit
    self.loss, sample_weight=sample_weight)
  File "/Strong/home/glickmanc/.conda/envs/py36/lib/python3.6/site-packages/sklearn/svm/_base.py", line 939, in _fit_liblinear
    epsilon, sample_weight)
  File "sklearn/svm/_liblinear.pyx", line 39, in sklearn.svm._liblinear.train_wrap
OverflowError: value too large to convert to npy_int32

```

My current script loads the table into a pandas dataframe object then does the following:

1. Sets target variable and removes column from matrix
2. Sets index using unique identifier
3. Scales columns using scikit-learn standard scalar
4. ```[Error Here]``` Performs lasso regression with LinearSVC ```[Error here]```
5. Selects columns used in modeling using SelectFromModel

I am getting the following error: OverflowError: value too large to convert to npy_int32

Any thoughts how to overcome this error would be appreciated. I have thought about converting the table to npy_int64, but am not sure how (already importing the dataframe using high float_precision and low_memory=False). 
user: glickmac
issue_18023: state: open
title: Add keyword parameter to scoring functions to support different types of data
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
Currently, there is no way to use `SelectKBest` with ordinal data. 

#### Describe your proposed solution
I would want to add a keyword parameter to the current `f_regression` (or `f_classif`) that takes in the type of input data. For example, if our `X` is ordinal and `y` is continuous, we can run `f_regression(X, y, input_type="ordinal")`. The function will then calculate the Spearman's coefficient (as opposed to the current implementation of Pearson's coefficient in `f_regression`) and output the scores and pvalues.

The steps to add support for ordinal data are:

- [ ] Write wrapper for `scipy.stats.spearmanr` OR write our own function that calculates Spearman's
- [ ] Integrate that wrapper into `f_regression` and add keyword parameter `input_type`

Now, I am not sure how to score one-hot encoded data yet, but hopefully by adding the keyword parameter, we can gradually expand the types of input data `sklearn`'s scoring functions can support.

#### Describe alternatives you've considered, if relevant
Alternatively, we can also write a new function `f_regression_ordinal` to deal with ordinal `X` and continuous `y`. 

#### Additional context
This feature request partially addresses https://github.com/scikit-learn/scikit-learn/issues/8480. There has also been discussions of the wrapper method, but no consensus has been reached: https://github.com/scikit-learn/scikit-learn/issues/6673, https://github.com/scikit-learn/scikit-learn/pull/8038.

This feature request was submitted per suggestions by @thomasjpfan and discussion with @yashika51 and @flosincapite
user: anhqngo
issue_18017: state: open
title: Implement scorers that evaluate return_std
body: While we don't tend to focus on uncertainty estimation on predictions, we support it through `return_std`, yet we do not have any scorers that take this into account for model selection and evaluation. Ideally where users want to make use of `return_std`, it should be considered as a factor in model quality.

Methods for measuring model performance while accounting for its prediction of uncertainty include:
* "mean standardized squared error" or "mean squared deviation ratio" (both terms used in geostatistics) defined as the mean over samples of the squared error divided by the predicted variance: 
![image](https://user-images.githubusercontent.com/78827/88611941-7a7b5e80-d0cd-11ea-908b-44e1d6c2aee4.png) (from [Webster and Oliver](https://onlinelibrary.wiley.com/doi/pdf/10.1002/9780470517277.ch8))). A median of this standardized squared error may also be used.
* Measuring the proportion of true values falling within a k% predictive interval. (Given that we only provide `return_std`, we need to assume gaussian residuals in any measure of error.)
user: jnothman
issue_18013: state: open
title: Incremental KernelPCA 
body: Hi everyone.

#### Describe the workflow you want to enable
When I run kernel-pca on my relatively large dataset I see the following behavior:
For 30 input features and radial kernel I get the following execution times:


- 10K rows: 5 minutes
- 20K rows: 6 minutes
- 30K rows: 52 minutes
- 50K rows: SIGKILL

Obviously I run out of memory during the execution, as the distance between every pair of rows is calculated. 

I am wondering if a kpca variant of incrementalPCA is possible

#### Describe your proposed solution
Implement `partial_fit` with appropriate adjustments so that it would be used the same way `partial_fit` is used in `IncrementalPCA`.


user: antonpuz
issue_18005: state: open
title: check_estimator and binary_only tag
body: Shouldn't the `check_estimator` check if you pass a non binary problem for a binary classifier? I mean, if you have an untagged binary estimator that raises an exception the `check_estimator` will fail. On the other hand,  if you have a tagged estimator as `binary_only` the `check_estimator` will pass if you pass a non-binary problem. I would expect to fail in this case.
user: chkoar
issue_18003: state: open
title: "probabilistic PCA" a misnomer
body: #### It is misleading to say the PCA method "implements the probabilistic PCA model".
<!--
Tell us about the confusion introduced in the documentation.
-->
The docs say
> Implements the probabilistic PCA model from: Tipping and Bishop (1999). "Probabilistic principal component analysis" ... http://www.miketipping.com/papers/met-mppca.pdf

but in fact the code <tt>_pca.py</tt> implements what those authors call "conventional PCA" (sec. 3.3, p. 6) inasmuch as the transforms use only the eigenvector matrix **U**, not the equations using the matrices **M** and **W**.
#### A potential alternative/fix
<!--
Tell us how we could improve the documentation in this regard.
-->
State that "probabilistic PCA" is used only to estimate the noise covariance *σ*² but not for the transforms because, as Tipping and Bishop state (p. 7), "there is no material benefit in so doing".
user: aimefournier
issue_17999: state: open
title: [WIP] POC for Factory-style construction of composite estimators
body: Implemented ColumnTransformer.append(), demonstrated in examples.

#### Reference Issues/PRs

Proof of concept demonstration of #16301

#### What does this implement/fix? Explain your changes.

This offers an alternative to choosing between listing ColumnTransformer transformers or Pipeline steps as tuples, and the simplified `make_*` invocation, by adding an `append()` method with explicit parameters and documentation.

user: jnothman
issue_17991: state: open
title: allowing string input for pairwise_distances
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs: Fixes  #15932
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: venkyyuvy
labels: Superseded: PR has been replace by a newer PR
issue_17978: state: open
title: predict_proba() for linear models with log loss can return NaNs with large model coefficients
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
The predict_proba() function can return NaNs for linear models with log loss if
the training has resulted in a model with large coefficients. The linear equation
can result in decision_function() returning large negative values for every class
for a single input. This means that the normalization in predict_proba_lr()
(line 327 in https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/linear_model/_base.py )
divides by zero, resulting in NaNs being returned.

I stumbled across this problem when using SGDClassifier with log loss and default
alpha. I wanted to train the model using minibatches and monitor the loss after
each batch (using metrics.log_loss and the outputs of predict_proba() ). I happened
to hit values of the model coefficients that created this issue. Increasing alpha to
increase regularization helps prevent the issue, but it would be good if it could be
avoided somehow, or if a more pertinent warning were given to the user.

#### Steps/Code to Reproduce
See gist: https://gist.github.com/richardtomsett/8b814f30e1d665fae2b4085d3e4156f5

#### Expected Results
predict_proba() returns a valid categorical probability distribution over the classes

#### Actual Results
predict_proba() returns NaN for some inputs

#### Versions
System:
    python: 3.8.3 (default, May 19 2020, 13:54:14)  [Clang 10.0.0 ]
executable: [redacted]
   machine: macOS-10.15.3-x86_64-i386-64bit

Python dependencies:
          pip: 20.0.2
   setuptools: 47.1.1.post20200604
      sklearn: 0.23.1
        numpy: 1.18.5
        scipy: 1.4.1
       Cython: None
       pandas: None
   matplotlib: 3.3.0
       joblib: 0.15.1
threadpoolctl: 2.1.0

Built with OpenMP: True

user: richardtomsett
issue_17976: state: open
title: Method to add Cost Function in F1 score 
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

The beta parameter determines the weight of recall in the combined score. beta < 1 lends
more weight to precision, while beta > 1 favors recall ( beta -> 0 considers only precision,
beta -> +inf only recall). This can be used to identify the best thresholds for the users to
get the best F-Score, considering we have a correct beta.

Though we consider the weights of recall and precision we often forget to introduce the cost
factor associated with Recall and Precision. Eg. Cost for Missing a rare and chronic disease
with poor Recall would be higher than having a bad precision.


#### proposed solution
I am thinking of an updated method that will introduce the cost function in the F1 score metrics.
The cost function can be varied from different case scenarios, 
one can deal with low precision but not low recall or vice versa. Here we can introduce cost functions for FP, FN, and TP, which users can introduce according to their needs and get a single value from it.

I am planning to introduce this methodology in an already established library, sci-kit learn. The
current function on will also adapt some capability from existing sklearn.metrics.fbeta_score
the functionality of this library.

#### alternatives 
There is a  research paper which can be tried 
1) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4442797/


user: siddhartha-shrivastava
issue_17975: state: open
title: Remove single-target assertion in LinearModelCV
body: Fixes #17965.

As far as I can tell this works just fine as-is; the only other change I made was removing the recently added `_is_multitask` since that seems no longer needed, and replacing with `_accepts_sparse` which before (as far as I can tell) was implicitly `not _is_multitask`.

I could maybe see an argument for keeping a warning but to be honest I think that's also overkill; I think adding a note to `ElasticNetCV` and `MultiTaskElasticNetCV` saying that one enforces the same sparsity pattern across both sets of coefficients while the other does not.

@drorcohengithub RidgeCV also seems to work fine as well FYI 🙂 
user: bnaul
issue_17965: state: open
title: Support multiple targets in LinearModelCV
body: I was reading the discussions in #13117 and #13131 and thought this point warranted its own issue:

> I thinks this could be improved, since both ElasticNet() and Lasso() support multiple targets. I don't see a reason why LinearModelCV() could not accept multiple targets as well.

I personally find the current behavior (namely rejecting inputs w/ multiple targets and referring the user to `MultiTask*CV` instead) to be a bit counterintuitive and undesirable: while there are certainly many cases where the MultiTask version is more appropriate, there are also cases where you explicitly *do not* want to share the sparsity pattern between the sets of coefficients. In that case you can currently use `MultiOutputRegressor(ElasticNetCV())` as a workaround, but it feels to me like supporting both cases directly would be a slightly more user-friendly option.

Happy to make a PR to remove the check in question (and resolve the other documentation issues as a bonus 🙂 ), just thought I'd try to get confirmation that I'm not overlooking anything here. cc @albertotb @agramfort @jnothman from the previous discussion
user: bnaul
issue_17962: state: open
title: EHN allow scorers to set addtional parameter of scoring function
body: closes #17942

Add a parameter copy to `get_scorer` to whether or not make a copy of a custom metric.
In addition, internal scikit-learn metrics will be copied.
user: glemaitre
issue_17960: state: open
title: Single linkage option in Agglomerative causes MemoryError for very large numbers
body: #### Describe the bug
Using the single linkage option in Agglomerative clustering results in a MemoryError when the input data contains very large values, likely due to numeric overflows. 

#### Steps/Code to Reproduce
Example: 
Running the code below with half of the sample data does not produce a MemoryError (although the same numpy RuntimeWarning is being thrown). However, if the data contains as many very large numbers as in my sample below my machine with 16Gb RAM runs out of memory and I get a MemoryError. To me this seems a bit disproportionate as the sample data is not very large.

```python
from sklearn.cluster import AgglomerativeClustering

X = [[1.30830774e+307, 6.02217328e+307],
     [1.54166067e+308, 1.75812744e+308],
     [5.57938866e+307, 4.13840113e+307],
     [1.36302835e+308, 1.07968131e+308],
     [1.58772669e+308, 1.19380571e+307],
     [2.20362426e+307, 1.58814671e+308],
     [1.06216028e+308, 1.14258583e+308],
     [7.18031911e+307, 1.69661213e+308],
     [7.91182553e+307, 5.12892426e+307],
     [5.58470885e+307, 9.13566765e+306],
     [1.22366243e+308, 8.29427922e+307],
     [4.39205961e+306, 1.26048413e+308],
     [4.61599953e+306, 7.24075646e+307],
     [1.66596896e+308, 1.65498552e+308],
     [4.73958815e+307, 7.66412710e+307],
     [1.57013390e+308, 1.03527051e+308],
     [1.28464631e+308, 1.68216358e+308],
     [1.07121506e+308, 3.11489418e+307],
     [2.97524276e+307, 1.59238260e+306],
     [1.24529964e+308, 5.18478922e+306],
     [7.55088957e+307, 7.08726240e+307],
     [8.38161061e+307, 1.50363727e+307],
     [1.11502738e+308, 3.77564517e+307],
     [1.33977566e+308, 4.21606136e+307],
     [3.18410347e+306, 1.41182675e+308],
     [1.37949806e+307, 1.33091975e+308],
     [8.84125355e+307, 8.83334687e+307],
     [4.16683700e+307, 4.38042780e+307],
     [6.33989592e+307, 1.13127438e+308],
     [3.30270525e+307, 5.82271903e+307],
     [1.73464450e+308, 1.57922601e+308],
     [8.50840567e+307, 8.34750980e+307],
     [1.06389805e+308, 7.01361194e+307],
     [1.20971776e+308, 1.42173002e+308],
     [1.91271271e+307, 7.63302091e+307],
     [1.46375293e+308, 3.27352625e+307],
     [5.64255485e+307, 1.62562194e+308],
     [5.96180877e+307, 4.31806469e+307],
     [1.18773200e+308, 4.91146568e+307],
     [1.19298612e+307, 1.54110558e+308],
     [3.51302340e+307, 5.60375139e+307],
     [5.14281492e+307, 9.76302343e+307],
     [1.77408023e+308, 1.65760854e+308],
     [1.46544741e+308, 1.47445640e+308],
     [1.76109403e+308, 1.30923042e+308],
     [1.64984146e+308, 2.58303609e+307],
     [1.61558758e+307, 1.03985868e+308],
     [1.37977676e+308, 4.90921157e+307],
     [1.01105745e+308, 1.57678709e+307],
     [1.24672794e+308, 5.96657664e+307]]

clusterer = AgglomerativeClustering(linkage='single')
clusterer.fit(X)
```

#### Expected Results
No error is thrown or ValueError that specifies the range of allowed values. GaussianMixture clustering is handling the same data like this:
```
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
```

#### Actual Results

```python-traceback
C:\Program Files\Python37\lib\site-packages\numpy\core\fromnumeric.py:90: RuntimeWarning: overflow encountered in reduce
  return ufunc.reduce(obj, axis, dtype, out, **passkwargs)

Traceback (most recent call last):
  File "C:/Users/thaar/PycharmProjects/sklearn-dev/agglomerative_test.py", line 58, in <module>
    clusterer.fit(X)
  File "C:\Program Files\Python37\lib\site-packages\sklearn\cluster\_agglomerative.py", line 898, in fit
    self.n_leaves_)
  File "C:\Program Files\Python37\lib\site-packages\sklearn\cluster\_agglomerative.py", line 674, in _hc_cut
    label[_hierarchical._hc_get_descendent(-node, children, n_leaves)] = i
  File "sklearn\cluster\_hierarchical_fast.pyx", line 96, in sklearn.cluster._hierarchical_fast._hc_get_descendent
MemoryError
```

#### Versions
System:
    python: 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\thaar\PycharmProjects\sklearn-dev\venv\Scripts\python.exe
   machine: Windows-10-10.0.18362-SP0

Python dependencies:
          pip: 20.1.1
   setuptools: 49.2.0
      sklearn: 0.23.1
        numpy: 1.18.4
        scipy: 1.4.1
       Cython: None
       pandas: 1.0.3
   matplotlib: 3.2.1
       joblib: 0.14.1
threadpoolctl: 2.0.0

Built with OpenMP: True

<!-- Thanks for contributing! -->

user: THaar50
issue_17954: state: open
title: Allow **fit_params on RFECV
body: Hi, 

minor change in `fit` method enabling extra fit parameters (i.e. `**fit_params`). 

Link to source code:
https://github.com/scikit-learn/scikit-learn/blob/fd237278e/sklearn/feature_selection/_rfe.py#L486

Thanks!
user: ciberger
labels: Hard: Hard level of difficulty
issue_17951: state: open
title: Provide a sentinel to detect unknown label in LabelPropagation
body: #### Describe the bug

For categorical labels, unlabeled data marked as '-1'  is recognized by the classifier as labelled

Say, 
**for integer labels=** 
labels = integer
-1 = interger

**for categorical labels=**
labels = string
'-1' = string
error = not recognized as unlabeled for this usecase!!!


#### Steps/Code to Reproduce

This is the example code with bug that needs to be fixed, line 234 and 245
```python

        classes = np.unique(y)

        classes = (classes[classes != -1])
        self.classes_ = classes

        n_samples, n_classes = len(y), len(classes)

        alpha = self.alpha
        if self._variant == 'spreading' and \
                (alpha is None or alpha <= 0.0 or alpha >= 1.0):
            raise ValueError('alpha=%s is invalid: it must be inside '
                             'the open interval (0, 1)' % alpha)
        y = np.asarray(y)
        unlabeled = y == -1



#### Expected Results
 Label_propagation.py should be able to recognize '-1' as unlabeled when passed as a string for categorical labels

#### Actual Results
 
Traceback can be seen here:
/home/ubuntu/.local/lib/python3.5/site-packages/sklearn/semi_supervised/label_propagation.py:234: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  classes = (classes[classes != -1])
/home/ubuntu/.local/lib/python3.5/site-packages/sklearn/semi_supervised/label_propagation.py:245: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  unlabeled = y == -1


#### Versions
>>> import sklearn; sklearn.show_versions()

System:
executable: /usr/bin/python3
   machine: Linux-4.4.0-aws-x86_64-with-Ubuntu-16.04
    python: 3.5.2 (default, Apr 16 2020, 17:47:17)

Python deps:
setuptools: 20.7.0
       pip: 19.3
    pandas: 0.24.2
     scipy: 1.3.1
    Cython: None
     numpy: 1.17.2
   sklearn: 0.21.3


>>> import sys; print("Python", sys.version)
Python 3.5.2 (default, Apr 16 2020, 17:47:17)

>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.17.2
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.3.1
>>> import sklearn; print("Scikit-Learn", sklearn.__version__)
Scikit-Learn 0.21.3
>>> import imblearn; print("Imbalanced-Learn", imblearn.__version__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'imblearn'


user: AartiKare
issue_17949: state: open
title: [WIP] Allow TruncatedSVD using randomized to automatically reset k < n_features.
body: #### Reference Issues/PRs
Fixes #17916

#### What does this implement/fix? Explain your changes.
Allows TruncatedSVD using the randomized algorithm to automatically reset k < n_features.

#### Any other comments?


user: zachmayer
issue_17930: state: open
title: [WIP] ENH create callable class to get adequate scorer for a problem
body: closes #17889

Alternative to #17889 
user: glemaitre
issue_17929: state: open
title: DOC Pilot of annotating parameters with category
body: 
#### Reference Issues/PRs

Resolves #17566 more or less.

#### What does this implement/fix? Explain your changes.

This demonstrates https://github.com/scikit-learn/scikit-learn/issues/17566#issuecomment-652929568 just for LogisticRegression.

It badges parameters as being either Control, Tuning or Resources parameters, per @mitar's [comment](https://github.com/scikit-learn/scikit-learn/issues/17566#issuecomment-652843664) regarding labels used for AutoML in the D3M project.

So far these labels add coloured badges, and link to glossary entries.

They are added in the docstring after a parameter's entire description, before `.. versionadded` etc., in a new paragraph. Is this the right place?
 
![image](https://user-images.githubusercontent.com/78827/87550155-17afbd80-c6f2-11ea-8efd-3f1b39dd361e.png)

#### Any other comments?

I think this is a minimal patch to demonstrate the idea, and then to invite contributors to label other parameters throughout the library. Extensions include:

* The labels should probably be positioned elsewhere (e.g. inset to the right of the parameter name), and this could be achieved with Javscript/DOM games.
* It would be nice to allow the user to only display parameters with a particular badge. This filtering again could be achieved with JavaScript.
* @mitar hoped for these annotations to be available as machine-readable metadata. We could achieve this with numpydoc.
user: jnothman
issue_17916: state: open
title: TruncatedSVD option to reduce k instead of raising "n_components must be < n_features;"
body: #### Describe the workflow you want to enable
If I'm using TruncatedSVD in a pipeline, it'd be nice to have an option to automatically set n_components < n_features, if n_components >= n_features.

For example, in the docs for [sklearn.manifold.TSNE](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html) suggest using TruncatedSVD to limit the dimensionality of the input to 50.  This is easy to do with a pipeline:

```python
from sklearn.decomposition import TruncatedSVD
from sklearn.manifold import TSNE
from sklearn.pipeline import make_pipeline
from scipy.sparse import random

tsne = make_pipeline(TruncatedSVD(n_components=50), TSNE(n_iter=250))

wide_data = random(1000, 100)
wide_tsne = tsne.fit_transform(wide_data)
```

However, let's say later we get a narrower dataset: 
```python
narrow_data = random(1000, 10)
narrow_tsne = tsne.fit_transform(narrow_data)
```

This raises the error: `n_components must be < n_features; got 50 >= 10`

#### Describe your proposed solution
I'd like to add a parameter to the `__init__` for TruncatedSVD, with a name like `excess_n_components`.  The default will be something like `excess_n_components="error"` which will preserve the current behaivor.

However, if `excess_n_components="reduce_n_components"` (or some other good way to specify it), at fit time we'd automatically reset `n_components` to `X.shape[1]-1`.  (With maybe a special case for when X.shape[1]==1?) 

#### Describe alternatives you've considered, if relevant
Writing a wrapper for Pipeline that generates different pipelines depending on the source data. This gets difficult; however, if the pipeline contains intermediate steps that may increase or decrease the dimensionality of the data.

#### Additional context

user: zachmayer
issue_17889: state: open
title: [WIP] ENH create a generator of applicable metrics depending on the target y
body: closes #12385
build on #15126

Create a generator of possible metrics to be used for a problem inferred from the target `y`.
The dictionary can be passed directly to the `GridSearchCV`.
Additional parameters can be passed to the utility which will be used when generating the metrics.
user: glemaitre
issue_17872: state: open
title: WIP option to cluster classes in confusion matrix
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
When there are many classes and you plot the confusion matrix; it is rather hard to interpret the results. The new feature clusters the classes based on how often one is misclassified for the other, the similar classes end up next to each other. This results in a clearer and easier to interpret plot.

#### Any other comments?
The problem, however, is that we will change the order of the classes, and the function normally only returns the confusion matrix without the labels. We choose to return the matrix as well as the labels, which is not in line with what happens now.
Suggestions on how to handle this differently are more than welcome.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: jochenater
issue_17871: state: open
title: Gaussian Process Regressor Kernel Default inaccurate
body: #### Describe the issue linked to the documentation

In the documentation of the [GPR](https://github.com/scikit-learn/scikit-learn/blob/93f0216c24c483ccd726bfd16b3951451bec29ae/sklearn/gaussian_process/_gpr.py#L46) it states that the kernel `"1.0 * RBF(1.0)"` is used as default when `None` is passed and that the hyperparameters of the kernel is fitted during the `fit` method. However, the default kernel (as seen [here](https://github.com/scikit-learn/scikit-learn/blob/93f0216c24c483ccd726bfd16b3951451bec29ae/sklearn/gaussian_process/_gpr.py#L181)) is instead `C(1.0, length_scale_bounds="fixed") * RBF(1.0, length_scale_bounds="fixed")`. This results in the described kernel differing from the actual default kernel, as the latter does not allow for parameter tuning.

#### Suggest a potential alternative/fix

There's two ways that a fix could be made:
- Describe clearly that the default kernel does not allow for hyperparameter tuning, either through changing the described kernel signature or implicit explanation.
- Change the default kernel to allow for hyperparameter tuning by removing the `length_scale_bounds` kwarg from at least the RBF kernel.

I use an issue to raise this rather than creating a merge request because I am not sure which of the above solutions is better. I personally think that it is better for the default behavior of the GPR to actually be fittable, which would require the second option to be chosen. However, I understand that people might disagree.

user: MaxVanDeursen
issue_17843: state: open
title: SubsampledNeighborsTransformer: Subsampled nearest neighbors for faster and more space efficient estimators that accept precomputed distance matrices
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
See #17650


#### What does this implement/fix? Explain your changes.
Instead of calculating the pairwise distances for all pairs of points to obtain nearest-neighbor graphs for estimators like DBSCAN, SubsampledNeighborsTransformer only calculates distances for a fraction s of the pairs (selected uniformly at random). This would make estimators that accept precomputed distance matrices feasible for larger datasets. In a very recent work I did with Google Research [1], we found that you can get over 200x speedup and 250x savings in memory this way without hurting the clustering quality (in some cases s = 0.001 suffices).

[1] https://arxiv.org/abs/2006.06743

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: jenniferjang
issue_17817: state: open
title: Add silhouette width using generalized mean
body: #### Describe the workflow you want to enable
According to https://scikit-learn.org/stable/modules/clustering.html#silhouette-coefficient

> The Silhouette Coefficient is generally higher for convex clusters than other concepts of clusters, such as density based clusters like those obtained through DBSCAN.

This seems to be true for all scores (available in sklearn) that are usable when ground truth labels are not known, and it could be an issue when you try to evaluate clusters obtained through algorithms such as DBSCAN and OPTICS.

#### Describe your proposed solution
In the following paper https://www.researchgate.net/publication/337370281_Silhouette_width_using_generalized_mean-A_flexible_method_for_assessing_clustering_efficiency there's a proposal to use the generalized mean (https://en.wikipedia.org/wiki/Generalized_mean) instead of the arithmetic mean to calculate average within‐ and between‐cluster distances. Changing the `p` parameter of the generalized mean (the arithmetic mean is a special case of the generalized mean with p=1), we can change the sensitivity of the index to compactness versus connectedness. This can be useful when we try to evaluate non-spherical clusters.

I know that this paper doesn't have 200+ citations, however it's a minor addition (it's a generalization of the silhouette width calculation), we can keep p=1 as default value for the `p` parameter to provide the current behavior, and it covers a scenario which is currently uncovered by sklearn (evaluation of non-convex clusters when ground truth labels are not available).

Is this something that we would like to have in sklearn?
user: candalfigomoro
issue_17816: state: open
title: Feature selector instances are too heavy-weight
body: ## Definitions
By "_selectors_", I'm referring to the set of classes that implement [`sklearn.feature_selection._base.SelectorMixin`](https://github.com/scikit-learn/scikit-learn/blob/fd237278e895b42abe8d8d09105cbb82dc2cbba7/sklearn/feature_selection/_base.py#L17). 

Examples are:
- `SelectKBest`
- `VarianceThreshold`.

## Summary

I'd like to suggest either:
1. modifying these "_selectors_" **to save less information in their state**
or 
2. providing a more minimal set of "_selectors_" **with less information saved in their state**

These classes save large matrices in their state which, for the most common cases, are not consumed.

## Motivation

_Consider a fairly typical NLP classifier as so:_
```python
Pipeline([
    ('feature-hashing', FeatureHasher(n_features=2**20)),
    ('variance', VarianceThreshold(threshold=0.0)),
    ('clf', LogisticRegression())
])
```

The `FeatureHasher` is described as "a low-memory alternative to DictVectorizer and CountVectorizer". The disadvantage of the `FeatureHasher` is that it will always produce a matrix with `n_features` regardless of the input size; therefore, using classes like `VarianceThreshold` and `SelectKBest` are commonly used to reduce the size of the feature matrix.

The current implementation of these classes mitigates much of the low-memory benefit of the FeatureHasher due to the large amount of information saved on their state.

For example:
- `VarianceThreshold` saves the matrix `variances_` of shape `(n_features,)` where n_features is the size of the input.
- `SelectKBest` saves two matrices with this shape: `scores_` and `pvalues_`

## Describe your proposed solution
I suggest that for this type of transformer, the state saved is a single numpy array of only the column indices to retain. An array in this form can be retrieved by calling `get_support(indices=True)` on a fitted selector instance.

- This dramatically reduces the size of the object (both when pickled and in-memory) without a decrease in algorithm performance.
- This implementation scales a lot better than the current implementation as the size of the state scales with the output shape rather than the input shape.

**The disadvantage of this approach is a decrease of model explain-ability**. 

To examine the difference of this approach, I designed the following implementation:

```python

class NumpyColumnFilter:

    def __init__(self, relevant_columns):
        """ Constructor.

        :param numpy.ndarray relevant_columns: 1D numpy array containing indicies of columns to
            retain.
        :return void:
        """
        self._relevant_columns = relevant_columns

    @classmethod
    def from_sklearn_selector(cls, selector):
        """ Produce a NumpyColumnFilter from an implementation of sklearn's SelectorMixin.

        :param mixed selector:
        :return NumpyColumnFilter:
        """
        return cls(selector.get_support(indices=True))

    def apply(self, X):
        """ Select just the relevant columns.

        :param numpy.ndarray X:
        :return numpy.ndarray:
        """
        return X[:, self._relevant_columns]


class FeatureSelector:

    def __init__(self, selector_class, **selector_kwargs):
        """ Constructor.

        :param mixed selector_class: Filter to remove feature indicies.
        :param dict selector_kwargs: Keyword arguments to pass when instantiating the selector class
        :return void:
        """
        self._selector_class = selector_class
        self._selector_kwargs = selector_kwargs
        self._column_filter = None

    def _is_fitted(self):
        """ Is the filter fitted?

        :return bool:
        """
        return self._column_filter is not None

    def fit_transform(self, *args, **kwargs):
        """ Fit and transform.

        :return np.ndarray:
        """
        return self.fit(*args, **kwargs).transform(*args, **kwargs)

    def fit(self, *args, **kwargs):
        """ Fit the algorithm.

        :param np.ndarray X:
        :raises Exception:
        :return self:
        """
        fitted_selector = self._selector_class(**self._selector_kwargs).fit(*args, **kwargs)

        self._column_filter = NumpyColumnFilter.from_sklearn_selector(fitted_selector)
        return self

    def transform(self, X, *args, **kwargs):
        """ Select just the relevant columns.

        :param np.ndarray X:
        :raises Exception:
        :return np.ndarray:
        """
        if not self._is_fitted():
            raise Exception('Not fitted!')

        return self._column_filter.apply(X)
```

### Comparing the size of the pickled objects show a dramatic decrease:

#### Setup code

```python
import pickle
import sys

import numpy as np
from sklearn import datasets
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import VarianceThreshold


def pickled_size(item):
    return sys.getsizeof(pickle.dumps(item))


X, y = datasets.make_classification(n_samples = 100, n_features = 10)

# Add one-million features without any variance.
X = np.concatenate((X, np.zeros((100, int(1e6)))), axis = 1)

```

#### `SelectKBest`

```python
sklearn_selector_class = SelectKBest().fit(X, y)
print(pickled_size(sklearn_selector_class))
>>> ~ 16 megabytes

feature_selector = FeatureSelector(selector_class = SelectKBest).fit(X, y)
print(pickled_size(feature_selector))
>>> ~ 500 bytes
```

#### `VarianceThreshold`

```python
sklearn_selector_class = VarianceThreshold().fit(X, y)
print(pickled_size(sklearn_selector_class))
>>> ~ 8 megabytes

feature_selector = FeatureSelector(selector_class = VarianceThreshold).fit(X, y)
print(pickled_size(feature_selector))
>>> ~ 400 bytes
```
user: DomHudson
issue_17815: state: open
title: Wrong equation for standard error of the mean
body: #### Describe the issue linked to the documentation

In the User Guide for [permutation importance](https://github.com/scikit-learn/scikit-learn/blob/fd237278e895b42abe8d8d09105cbb82dc2cbba7/doc/modules/permutation_importance.rst), the second code block uses `2 * r.importances_std[i]` to estimate 95% confidence interval. As far as I understand, instead of 2 * standard deviation it should be 2 * standard error of the mean. The standard error of the mean is calculated as the standard deviation divided by the square root of the sample size which is `r.importances.shape[1]` in this case. Moreover, the difference for degrees of freedom in the Numpy standard deviation by default is `ddof=0` while the most common definition of the standard deviation uses `ddof=1` ("corrected sample standard deviation").

#### Suggest a potential alternative/fix

1. If the intention was to return the standard error of the mean (which makes more sense to me), then I would recommend to replace the returned value [`importances_std`](https://github.com/scikit-learn/scikit-learn/blob/fd237278e895b42abe8d8d09105cbb82dc2cbba7/sklearn/inspection/_permutation_importance.py#L141) to `importances_sem=scipy.stats.sem(importances, axis=1)`. Note that the `scipy.stats.sem()` by default uses `ddof=1` as mentioned above.

2. If the intention was to return the standard deviation, then the equation in the documentation should be `2 * r.importances_std[i] / np.sqrt(r.importances.shape[1])` and the difference for degrees of freedom in [`importances_std`](https://github.com/scikit-learn/scikit-learn/blob/fd237278e895b42abe8d8d09105cbb82dc2cbba7/sklearn/inspection/_permutation_importance.py#L141) should be adjusted like this `importances_std=np.std(importances, axis=1, ddof=1)`.

user: PavloFesenko
issue_17813: state: open
title: Improving Mahalanobis distance KNN speed
body: <!--
I understand this proposition may not fit the inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms

In this case, please disregard the proposal.
-->

#### Describe the workflow you want to enable:

KNN usage with Mahalanobis can become rather slow (several seconds per test datapoint) when the feature space is large (1500 features). Even if the training set is small (100s of images)

#### Describe your proposed solution:

Mahalanobis distance computes d = (x-y)T VI (x-y) for each x in the training set.

If we develop we get d = xT.VI.x + yT.VI.y - xT.VI.y - yT.VI.x
Now the first term could be computed at init_time for all x in training set, as well as xT.VI and VI.x.

This way, at test time:
- yT.VI.y could be computed only once. It is not necessary to compute it for all x.
- for each x in training set, only two dot products are left to compute the distances. 

The issue with this is now we cannot have fully parallel (in terms of training set elements) code of distance computation in the KNN implementation.

#### Describe alternatives you've considered, if relevant

Similar idea can be found here: https://www.ece.ucsb.edu/scl/html/database/RelevanceFeedbackIndexing_ICIP2008.pdf


user: PGuti
labels: Moderate: Anything that requires some knowledge of conventions and best practices
Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_17806: state: open
title: MNT Deprecates _estimator_type and replaces by a estimator tag
body: #### Reference Issues/PRs

Closes #16469.

#### What does this implement/fix? Explain your changes.

This PR deprecates the `_estimator_type` private attribute, replaced by a newer estimator tag.

user: alfaro96
issue_17799: state: open
title: ENH Adds typing support to LogisticRegression
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Related to https://github.com/scikit-learn/scikit-learn/issues/16705
Related to https://github.com/scikit-learn/scikit-learn/issues/11170


#### What does this implement/fix? Explain your changes.
This PR experiments with static typing by:

1. Adding typing to parameters and attributes of `LogisticRegression`.
2. Uses `Annotated` from [PEP 593](https://www.python.org/dev/peps/pep-0593/) to provide metadata about the shape of a `ndarray`.
3. 
Adds a `inject_docstring` keyword to `__init_subclass__` to inject the type into the docstring. This allows us to not depend on numpydoc. We also have custom formating when we define types in our docstrings.

4. Another option is to have the type only in the generated docs so we would not need to inject the types during runtime.
5. This will help standardize the types in our docstrings since they are automatically generated.

#### Any other comments?

This PR is for adding simple types:

```py
    def __init__(self,
                  penalty: Literal['l1', 'l2', 'elasticnet', 'none'] = 'l2',
                  *,
                  dual: bool = False,
                  tol: float = 1e-4,
                  C: float = 1.0,
                  fit_intercept: bool = True,
                  intercept_scaling: float = 1,
                  class_weight: Union[dict, Literal['balanced']] = None)
```

**Not part of this PR and the future** I can see this `Annotated` be useful with ranges:

```py
max_features: Union[Annotated[float, Range(0, 1)], Annotated[int, Range(1, upper=None)]],
                    Literal['auto', 'sqrt', 'log2'], None]
```

CC @rth

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
issue_17798: state: open
title: [0.23.1] AdaBoostRegressor failes on arm(rhel) processors
body: #### Describe the bug
On arm processors (AWS:gravition2, rhel), I get the following failure in version 0.23.1
```
937     Examples
938     --------
939     >>> from sklearn.ensemble import AdaBoostRegressor
940     >>> from sklearn.datasets import make_regression
941     >>> X, y = make_regression(n_features=4, n_informative=2,
942     ...                        random_state=0, shuffle=False)
943     >>> regr = AdaBoostRegressor(random_state=0, n_estimators=100)
944     >>> regr.fit(X, y)
945     AdaBoostRegressor(n_estimators=100, random_state=0)
946     >>> regr.predict([[0, 0, 0, 0]])
Expected:
    array([4.7972...])
Got:
    array([5.57874246])

/home/ec2-user/github/sklearn_master/sklearn/ensemble/_weight_boosting.py:946: DocTestFailure
```

#### Steps/Code to Reproduce
`pytest -v sklearn/ensemble/_weight_boosting.py::sklearn.ensemble._weight_boosting.AdaBoostRegressor`

#### Expected Results
PASSED is thrown. 

#### Actual Results
FAILED is thrown.
```
946     >>> regr.predict([[0, 0, 0, 0]])
Expected:
    array([4.7972...])
Got:
    array([5.57874246])
```

#### Versions
```
System:
    python: 3.6.8 (default, Dec  5 2019, 16:02:25)  [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
executable: /usr/bin/python3
   machine: Linux-4.18.0-193.1.2.el8_2.aarch64-aarch64-with-redhat-8.2-Ootpa

Python dependencies:
          pip: 20.1.1
   setuptools: 39.2.0
      sklearn: 0.23.1
        numpy: 1.14.3
        scipy: 1.0.0
       Cython: 0.29
       pandas: 1.0.5
   matplotlib: 3.2.1
       joblib: 0.14.0
threadpoolctl: 2.1.0

Built with OpenMP: True
Linux-4.18.0-193.1.2.el8_2.aarch64-aarch64-with-redhat-8.2-Ootpa
Traceback (most recent call last):
  File "<string>", line 3, in <module>
NameError: name 'Python' is not defined
```
user: murata-yu
labels: arch:arm: ARM related issues
issue_17795: state: open
title: Labeller action: "PR welcome"
body: Would it be useful to have an automatic labeller for issues that sees the words "PR welcome" or "pull request welcome" in a comment by a Triage or Core Dev team member? It would then relabel a "Bug: triage" issue as "Bug" and add "help wanted".

user: jnothman
labels: workflow: Development workflow changes
issue_17782: state: open
title: RFECV to provide the average ranking_ and support_
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
Currently, `RFECV` searches for the optimal number of features to provide the best score. It then fits an `RFE` on the whole training set: [L566](https://github.com/scikit-learn/scikit-learn/blob/fd237278e/sklearn/feature_selection/_rfe.py#L566)

So, the score is the cross-validated score, but the `ranking_` is for the whole set of data. Isn't this misleading? e.g. with `RFE` and 3-fold cross-validation, I get these `ranking_`s:
```
[1 1 4 1 1 2 3 5 7 6]
[1 1 2 1 1 7 4 3 5 6]
[1 1 2 1 1 7 5 6 3 4]
```
while `RFECV` returns something like:
```
[1 1 3 1 1 5 2 4 7 6]
```
which is the `ranking_` `RFE` would provide if fit over the whole set. So, the score is a cross-validated score, while the `ranking_` and `support_` are not.

#### Describe your proposed solution
Imagine a situation where you get the best performance with `k` features, but the `k` features in some folds are different from the `k` features you would get by fitting the model on the whole training set. Wouldn't it be a better idea to return a form of average/voting_based `ranking_` and `support_`? 
user: apptimise
issue_17775: state: open
title: ENH add interpolation parameter to DummyRegressor for "median" and "quantile" strategies
body: It requires merging #17768

This PR provides the following:

* expose an `interpolation` parameter which can be used with the `median` and `quantile` strategies;
* no change of the default behaviour;
* raise a FutureWarning to have consistent `interpolation` when passing or not `sample_weight`. `"linear"` will be the default.

This PR is useful for creating the original estimator in the GBRT where we want the same interpolation with and without `sample_weight` meaning that we want to set it. 
user: glemaitre
issue_17768: state: open
title: ENH improve _weighted_percentile to provide several interpolation
body: closes #6189
closes #7662

Toward the completion of  #17377
Next step #17775

Implement different interpolation strategies for `_weighted_percentile`

user: glemaitre
issue_17748: state: open
title: Indicating None as an option in docstrings of n_jobs and random_state
body: For `random_state` and `n_jobs`, `None` has a very specific meaning so I think we should explicitly mention it as an option in the docstring:

`random_state : int, RandomState instance or None, default=None`

and 

`n_jobs : int or None, default=None`

We should not just write ``n_jobs : int, default=None`` because this suggests that the default is that there is no jobs.

The meaning of `None` for `random_state` and `n_jobs` is very different from the meaning of `None` for e.g. `sample_weight=None` so I don't think we should treat it the same.
user: NicolasHug
issue_17744: state: open
title: [WIP] NEP-18 support for preprocessing algorithms
body: **Reference Issues/PRs**

This PR completes the existing experimental attempt to support NEP-18, see #16196

**What does this implement/fix? Explain your changes.**

- Modify `create_like` to support CuPy arrays
- Modify a few parts of some preprocessing algorithms for NEP-18 support
- Add CuPy tests for some preprocessing algorithms
user: viclafargue
issue_17711: state: open
title: Implement probability estimates for NearestCentroid classifier
body: #### Reference Issues/PRs
There are no related issues or PRs.

#### What does this implement/fix? Explain your changes.
This contribution implements the .predict_proba() method for the NearestCentroid() classifier based on the contribution of 
_Tibshirani, R., Hastie, T., Narasimhan, B., & Chu, G. (2002). Class Prediction by Nearest Shrunken Centroids, with Applications to DNA Microaarrays. Statistical Science 18(1), p. 104-117._


#### Any other comments?
Currently, the estimation of probabilities is optional and has to be enabled at the instantiation of the classifier. However, this is a design decision and any advise is welcome.

user: kempa-liehr
issue_17684: state: open
title: PERF Optimize dot product order
body: When multiplying 3 or more matrices, the order of parathesis doesn't impact the results but it can have a very significant impact on the number of operations and on performance see https://en.wikipedia.org/wiki/Matrix_chain_multiplication

For matrix multiplication of dense arrays there is [numpy.linalg.multi_dot](https://numpy.org/doc/stable/reference/generated/numpy.linalg.multi_dot.html) and we we should use it I think. To find existing occurrences where it could be used, see for instance the result of
```
git grep 'dot(.*dot'
```

```
sklearn/datasets/_samples_generator.py:    return np.dot(np.dot(u, s), v.T)
sklearn/datasets/_samples_generator.py:    X = np.dot(np.dot(U, 1.0 + np.diag(generator.rand(n_dim))), Vt)
sklearn/decomposition/_fastica.py:    w -= np.dot(np.dot(w, W[:j].T), W[:j])
sklearn/decomposition/_fastica.py:    return np.dot(np.dot(u * (1. / np.sqrt(s)), u.T), W)
sklearn/decomposition/_fastica.py:                S = np.dot(np.dot(W, K), X).T
sklearn/decomposition/_nmf.py:            norm_WH = trace_dot(np.dot(np.dot(W.T, W), H), H)
sklearn/decomposition/_nmf.py:        denominator = np.dot(np.dot(W.T, W), H)
sklearn/discriminant_analysis.py:        self.coef_ = np.dot(self.means_, evecs).dot(evecs.T)
sklearn/gaussian_process/_gpc.py:            s_1 = .5 * a.T.dot(C).dot(a) - .5 * R.T.ravel().dot(C.ravel())
sklearn/gaussian_process/_gpc.py:            s_3 = b - K.dot(R.dot(b))  # Line 14
sklearn/linear_model/_bayes.py:            coef_ = np.dot(X.T, np.dot(
sklearn/linear_model/_logistic.py:        ret[:n_features] = X.T.dot(dX.dot(s[:n_features]))
sklearn/linear_model/_ridge.py:        AXy = A.dot(X_op.T.dot(y))
```

Ideally each replacement should be benchmarked.


For matrix multiplication `safe_sparse_dot` using a combination of sparse and dense matrice, some of this could apply as well, though defining a general heuristic is probably a bit more difficult there.
user: rth
issue_17676: state: open
title: [WIP] PCA NEP-37 adding random pathway and CuPy test
body: **Reference Issues/PRs**

This PR completes the existing experimental attempt to enable NEP-37 for the PCA algorithm.
See #16574

**What does this implement/fix? Explain your changes.**

- Implement the pathway that make use of `randomized_svd` when `svd_solver='randomized'`
- Add a CuPy test
user: viclafargue
issue_17675: state: open
title: pandas dtypes "boolean" not supported in classification target
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
Pandas has extended NumPy's dtypes and these dtypes extensions are not all supported as targets in a sklearn classifier. In particular, if the target y is a Pandas "boolean" dtype, a classifier such as LogisticRegression fails whereas if the target is a numpy "bool" dtype, the classifier will not fail.

#### Steps/Code to Reproduce
```python
import pandas as pd
from sklearn.linear_model import LogisticRegression

df = pd.DataFrame({"boolean_col": pd.Series([False, True, False, True], dtype="boolean"),
                   "bool_col": pd.Series([False, True, False, True], dtype="bool"),
                   "num_col": pd.Series([1, 2, 3, 4])})

clf = LogisticRegression()
```




#### Expected Results
```python
clf.fit(df[["num_col"]], df.bool_col)
--------------------------------------------------------------------------------------------------------------
LogisticRegression()
```
#### Actual Results
```python
clf.fit(df[["num_col"]], df.boolean_col)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-9a0c5abf260e> in <module>
----> 1 clf.fit(df[["num_col"]], df.boolean_col)

~\AppData\Local\Continuum\anaconda3\envs\lognode\lib\site-packages\sklearn\linear_model\_logistic.py in fit(self, X, y, sample_weight)
   1343                                    order="C",
   1344                                    accept_large_sparse=solver != 'liblinear')
-> 1345         check_classification_targets(y)
   1346         self.classes_ = np.unique(y)
   1347 

~\AppData\Local\Continuum\anaconda3\envs\lognode\lib\site-packages\sklearn\utils\multiclass.py in check_classification_targets(y)
    170     if y_type not in ['binary', 'multiclass', 'multiclass-multioutput',
    171                       'multilabel-indicator', 'multilabel-sequences']:
--> 172         raise ValueError("Unknown label type: %r" % y_type)
    173 
    174 

ValueError: Unknown label type: 'unknown'
```
#### Versions
System:
    python: 3.8.2 (default, Apr 14 2020, 19:01:40) [MSC v.1916 64 bit (AMD64)]
executable: ~\AppData\Local\Continuum\anaconda3\envs\lognode\python.exe
   machine: Windows-10-10.0.18362-SP0

Python dependencies:
          pip: 20.0.2
   setuptools: 46.1.3.post20200330
      sklearn: 0.23.1
        numpy: 1.18.1
        scipy: 1.4.1
       Cython: None
       pandas: 1.0.3
   matplotlib: 3.2.1
       joblib: 0.14.1
threadpoolctl: 2.1.0

Built with OpenMP: True
user: yasmina-altair
issue_17660: state: open
title: [Feature Request] Arbitrary base learners for GradientBoostingRegressor/Classifier
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable / Describe your proposed solution
add an `base_estimator` argument to GradientBoostingRegressor/Classifier and HistGradientBoostingRegressor/Classifier, allowing them to do gradient boosting on any sklearn base estimator that supports weights, while also supporting GradientBoostingRegressor like early stopping, custom objective functions, and early stopping.

#### Describe alternatives you've considered, if relevant
[sklearn.ensemble.AdaBoostClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html) supports specifying an arbitrary base_estimator It'd be pretty cool to have that for GradientBoostingRegressor/Classifier and HistGradientBoostingRegressor/Classifier

#### Additional context

user: zachmayer
issue_17658: state: open
title: Inconsistency between coef_ and coef_path_ in LassoLars
body: #### Describe the issue linked to the documentation

Hello, the coef_ attribute from LassoLars returns the array of coefficients for the value of alpha given in parameter (1 by default). The coef_path_attribute is described in the doc as follow : "The varying values of the coefficients along the path".
However, **the last column of the coef_path_ doesn't match the values of the coef_ attribute**, whereas the alpha value for this column is corresponding to the one given in parameter, hence the values of coefficients should be the same.

For example, if I run :
```python
from sklearn.linear_model import LassoLars
from sklearn.datasets import make_regression

X, y = make_regression(n_samples=50,
                       n_features=10,
                       n_informative=3)

lasso = LassoLars().fit(X, y)
```
coef_ will look like :
```
array([ 0.        , 39.89632517, 77.75976515, 88.68204422,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ])
```
and coef_path_ :
```
array([[  0.        ,   0.        ,   0.        ,   0.        ],
       [  0.        ,   0.        ,   0.        , 289.86068748],
       [  0.        ,   0.        , 122.80186215, 573.97268871],
       [  0.        , 123.33792921, 246.13979136, 584.906583  ],
       [  0.        ,   0.        ,   0.        ,   0.        ],
       [  0.        ,   0.        ,   0.        ,   0.        ],
       [  0.        ,   0.        ,   0.        ,   0.        ],
       [  0.        ,   0.        ,   0.        ,   0.        ],
       [  0.        ,   0.        ,   0.        ,   0.        ],
       [  0.        ,   0.        ,   0.        ,   0.        ]])
```
#### Suggest a potential alternative/fix

Either normalize the 2 attributes in the same way, or give additional information about the coef_path_ values in the doc.


user: FloWPs
issue_17655: state: open
title: GridSearchCV's fir function doesn't accept a list of inputs
body: I have a Siamese Neural Network which requires two inputs and in Keras this is allowed by passing a list of inputs in the fit function, however when I tried to do a grid search on my model I got a Value Error as GridSearchCV doesn't seem to accept multiple inputs in the fit function.

This means I'm not able to use grid search on any model with multiple inputs and therefore I would suggest adding such a feature to the fit function so any model created with keras could be used.

user: AliRSajun
issue_17650: state: open
title: DBSCAN too slow and consumes too much memory for large datasets: a simple tweak can fix this.
body: #### Describe the workflow you want to enable

Currently, DBSCAN is very slow for large datasets and can use a lot of memory, especially in higher dimensions. 

For example, running sklearn.cluster.DBSCAN on a size 745k dataset with 411 features took over 6 days to finish running on a powerful cloud machine, and running on a size ~1M dataset with a reasonable setting of epsilon for that dataset ran out of memory on that machine with over 750GB RAM.

In a very recent work I did with Google Research [1], we found that you can get over 200x speedup and 250x savings in memory based on a minor tweak to the DBSCAN algorithm without sacrificing clustering accuracy.

[1] https://arxiv.org/abs/2006.06743

#### Describe your proposed solution

The solution is based on a simple modification to the brute force version. Instead of calculating the pairwise distances for all pairs of examples to obtain the nearest-neighbor graph, you can instead only calculate it for a fraction s of the pairs (selected randomly) and clusters based on that subsampled graph. This would make the algorithm feasible for larger datasets, without hurting the clustering quality (in some cases s = 0.001 suffices). The idea is that you don’t actually need the full nearest-neighbor graph to get the desired clustering.

All comparisons in [1] were done against the algorithm=’auto’ setting of sklearn.cluster.DBSCAN.

The only new parameter needed would be 0 < s <= 1, the sampling ratio which would only be applicable for the algorithm=’brute-force’ setting. (Perhaps we might also want a random seed parameter if one wants to be able to have reproducible runs).

#### Describe alternatives you've considered, if relevant

#### Additional context
user: jenniferjang
issue_17642: state: open
title: Tutorial on how to create a custom kernel for Gaussian Process
body: #### Reference Issues/PRs
Fixes #9601. Continues the work of https://github.com/scikit-learn/scikit-learn/pull/9605

#### What does this implement/fix? Explain your changes.
Add tutorial on how to create a custom kernel for Gaussian Process.

#### Any other comments?
user: anhqngo
issue_17631: state: open
title: Using make_scorer() for a GridSearchCV scoring parameter in a clustering task
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### * Workflow:
1- Consider `make_scorer()` below for a **clustering** metric:
```python
from sklearn.metrics import homogeneity_score, make_scorer

def score_func(y_true, y_pred, **kwargs):
    return homogeneity_score(y_true, y_pred)
scorer = make_scorer(score_func)
```
2- Consider the simple method **optics()**:
```python
# "optics" algorithm for clustering
# ---
def optics(data, labels):
    # data: A dataframe with two columns (x, y)
    preds = None    
    base_opt = OPTICS()
    grid_search_params = {"min_samples":np.arange(10),
                          "metric":["cityblock", "cosine", "euclidean", "l1", "l2", "manhattan"],
                          "cluster_method":["xi", "dbscan"],
                          "algorithm":["auto", "ball_tree", "kd_tree", "brute"]}
    
    grid_search_cv = GridSearchCV(estimator=base_opt,
                                  param_grid=grid_search_params,
                                  scoring=scorer)
    
    grid_search_cv.fit(data)    
    opt = grid_search_cv.best_estimator_
    opt.fit(data)
    preds = opt.labels_
    
    # return clusters corresponding to (x, y) pairs according to "optics" algorithm
    return preds
```
Running the `optics()` led to this error:
**`TypeError: _score() missing 1 required positional argument: 'y_true'`**

Even by using `grid_search_cv.fit(data, labels)` instead of `grid_search_cv.fit(data)`, another exception rised:
**`AttributeError: 'OPTICS' object has no attribute 'predict'`**

---

**_I thinks we cannot use `make_scorer()` with a `GridSearchCV` for a clustering task._**

---
#### * Proposed solution:
The `fit()` method of `GridSearchCV` automatically handles the type of the estimator which passed to its constructor, for example, for a clustering estimator it considers `labels_` instead of `predict()` for scoring.

user: imanirajian
issue_17626: state: open
title: PERF Consider using argpartition in ndcg_score
body: As reported by @karlhigley,

> I now take issue with the implementation of NDCG in sklearn, which seems like it could use `argpartition` and be much faster for long lists of items with small top-K results (e.g. NDCG@100 with 60000 items.)

Would you like to propose a PR to improve it?
user: rth
issue_17625: state: open
title: inconsistent treatment of None and np.NaN in SimpleImputer
body: Doing constant imputation treats only the "missing_value" as missing, so a ``None`` by default stays there:
```python
from sklearn.impute import SimpleImputer
import numpy as np
X = np.array([1, 2, np.NaN, None]).reshape(-1, 1)
SimpleImputer(strategy='constant', fill_value="asdf").fit_transform()
```
```
array([[1],
       [2],
       ['asdf'],
       [None]], dtype=object)
```
However, using strategy='mean' coerces the None to NaN and so both are replaced:
```python
SimpleImputer(strategy='mean').fit_transform(X)
```
```
array([[1. ],
       [2. ],
       [1.5],
       [1.5]])
```
I don't think the definition of what's missing should depend on the strategy. @thomasjpfan argues that the current constant behavior is inconvenient because it means you have to impute both values separately if you want to one-hot-encode.

It seems more safe to treat them differently but I'm not sure there's a use-case for that. This came up in #17317.
I think this only matters in these two, as other imputers don't allow dtype object arrays.
user: amueller
issue_17624: state: open
title: BayesianRidge fails when input and output data are of very different sizes
body: <!--
Unlike LinearRegression, Bayesian Ridge does not  behave reliably. Here is an example comparison between the two.
-->

Example 1 (working):
```python
from sklearn.linear_model import LinearRegression, BayesianRidge
import numpy as np

ols = LinearRegression()
ols.fit(np.reshape([1,2],(-1,1)), np.array([2,3]).ravel())
clf = BayesianRidge(compute_score=True, fit_intercept=False)
clf.fit(np.array([[1,1],[1,2]]), np.array([2,3]).ravel())

print(str(ols.intercept_) + " " + str(ols.coef_[0]))
print(str(clf.coef_[0]) + " " + str(clf.coef_[1]))
```
#### Expected Results
1,1

#### Results for OLS and BayesianRidge:
1.0000000000000004  0.9999999999999998
0.9988917252390923  1.0005536752418909



Example 2 (not working):
```python
from sklearn.linear_model import LinearRegression, BayesianRidge
import numpy as np

ols = LinearRegression()
ols.fit(np.reshape([1,2],(-1,1)), np.array([2000000,3000000]).ravel())
clf = BayesianRidge(compute_score=True, fit_intercept=False)
clf.fit(np.array([[1,1],[1,2]]), np.array([2000000,3000000]).ravel())

print(str(ols.intercept_) + " " + str(ols.coef_[0]))
print(str(clf.coef_[0]) + " " + str(clf.coef_[1]))
```
#### Expected Results
1000000, 1000000

#### Results for OLS and BayesianRidge:
1000000.0000000005 999999.9999999997
7.692319353001738e-07 1.2307710964802638e-06


Please notice that the only difference betweenn the two examples are the order of magnitude of the endogenous variable! However, although OLS works pretty well, in the second case we get 0,0 as the coefficients from the Bayes regression"



user: SB6988
issue_17575: state: open
title: escape strings for export_graphviz
body: #### Reference Issues/PRs


#### What does this implement/fix? Explain your changes.
Double quotes in graphviz labels need escaping. Escape class and feature
names to avoid breaking the result.

#### Any other comments?


user: smola
issue_17566: state: open
title: API split "model" parameters from "config" params.
body: We can think of our estimators' parameters as two different types:
- model hyper-parameters which influence the end result, accuracy, complexity, etc., e.g. SVM's `nu` or the number of trees in a random forest.
- config parameters which control other aspects of an estimator such as verbosity, copy_X, return_xyz, etc.

We could split these two types of parameters and have _model hyperparams_ as `__init__` args and the rest as `set_params` or a new `configure` args. This would also allow us to introduce new configuration parameters to all estimators w/o having to add another arg to `__init__`. An example would be the routing parameter required for sample props implementation, or potentially controlling log level at the estimator level.

My proposal would be to have a `config_params` estimator tag which includes the names of those parameters, and have `clone`, `get_params` and `set_params` understand them, and potentially introduce a new `configure` method to set those parameters.

This also would mean partially relaxing the _set only `__init__` params in `__init__`_ requirement since `__init__` would be allowed to set those extra params.

cc: @scikit-learn/core-devs 

also related to: https://github.com/scikit-learn/scikit-learn/pull/17441#discussion_r435447166
user: adrinjalali
issue_17563: state: open
title: KBinsDiscretizer.inverse_transform should have different options
body: #### Describe the workflow you want to enable

KBinsDiscretizer.inverse_transform uses the middle point of the edge ranges.
However, for discrete values this is poor, and we would prefer to pick the median value
of points trained in the range, and perhaps also the mode. This could be enabled with a
parameter `bin_strategy='median'` either in the constructor or in `inverse_transform`

```
import numpy as np
import sklearn.preprocessing as preprocessing
from sklearn.feature_extraction import DictVectorizer
jsons = [ {"attack": 0.1}, {"attack": 1.0} ]
v = DictVectorizer(sparse=False)
X = v.fit_transform(jsons)
v.inverse_transform(X)
```
gives
```
[{'attack': 0.1}, {'attack': 1.0}]
```

```
est = preprocessing.KBinsDiscretizer(n_bins=3, encode='onehot', strategy='uniform').fit(X)
v.inverse_transform(est.inverse_transform(est.transform(X)))
```
gives
```
[{'attack': 0.25}, {'attack': 0.85}]
```

So we have *lost* the original values through binning, through weird choice of edges.

However, we would prefer:
```
v.inverse_transform(est.inverse_transform(est.transform(X), bin_strategy='median'))
```

gives
```
[{'attack': 0.1}, {'attack': 1.0}]
```
user: txntxn
issue_17557: state: open
title: Linear SVM does wrong prediction
body: #### Describe the bug
The prediction for specific train data is wrong

#### Steps/Code to Reproduce
You can run the following code. While the first decision limit is correct, for the 2nd pair of points the decision limit is totally wrong placed

Example:
```python
from sklearn import svm
from matplotlib import pyplot as plt
import numpy as np

lclf = svm.LinearSVC()

p1 = 0.25
p2 = -0.25

x_min = -2
x_max = 2
x_h = 0.01
y_min = -2
y_max = 2
y_h = 0.01
xx, yy = np.meshgrid(np.arange(x_min, x_max, x_h),
                    np.arange(y_min, y_max, y_h))

lclf.fit([[p1,0],[p2,0]],[-1,1])

plt.plot(p1, 0, "ro")
plt.plot(p2, 0, "go")

Z = lclf.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

plt.gca().contourf(xx, yy, Z, colors = ["red", "green"], alpha = 0.1)
plt.show()


#The following does not work

p3 = 0.025545042466768184
p4 = 0.0293294646367189

x_min = 0
x_max = 0.035
x_h = 0.00001
y_min = -0.01
y_max = 0.01
y_h = 0.001
xx, yy = np.meshgrid(np.arange(x_min, x_max, x_h),
                    np.arange(y_min, y_max, y_h))

lclf.fit([[p3, 0], [p4,0]], [1,-1])

plt.plot(p3, 0, "go")
plt.plot(p4, 0, "ro")

Z = lclf.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

plt.gca().contourf(xx, yy, Z, colors = ["red", "green"], alpha = 0.1)
plt.show()
```


#### Expected Results
The decision boarder should be centered between the two train points

#### Actual Results
the boarder is on the left of both points

#### Versions
System:
    python: 3.7.5 (tags/v3.7.5:5c02a39a0b, Oct 15 2019, 00:11:34) [MSC v.1916 64 bit (AMD64)]
executable: C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\python.exe
   machine: Windows-10-10.0.18362-SP0

Python dependencies:
       pip: 20.0.2
setuptools: 46.1.3
   sklearn: 0.22.2.post1
     numpy: 1.18.3
     scipy: 1.4.1
    Cython: None
    pandas: 1.0.3
matplotlib: 3.2.1
    joblib: 0.14.1

Built with OpenMP: True

user: C4rst3n
issue_17544: state: open
title: [WIP] Use dataset factories for estimator checks
body: ### Reference Issues/PRs

This is a follow-up of #14057 


#### What does this implement/fix? Explain your changes.

The idea here is to provide a dataset factory to the estimator checks. This would allow downstream packages to define their own dataset factory to check their estimators on the specific kind of data they expect as input.

#### Any other comments?

This first commit is a small step towards implementing @rth 's suggestion, just to make sure that I did not misunderstand his suggestion.
The idea would be to use the dataset factory for all `check_*` functions (or at least for as many functions as possible).
user: rtavenar
issue_17543: state: open
title: Python hangs while exploring a GridSearCV with n_jobs=-1
body: Dear all,

I have been encountering the issue mentioned in the description for a while and as the [10533](https://github.com/scikit-learn/scikit-learn/issues/10533) where the issue is discussed does not present a clear **solution for Windows**, I decided to post it here.

Firstly, **the piece of code that runs in two the issue** is the following:
`    qrf = RandomForestQuantileRegressor(n_jobs=-1)`

    parameters = {'n_estimators' : [30,40], # More estimators will always give a better result
                  'criterion': ['mae'],
                  'min_samples_split': [5,10,15],
                  'max_features' : [Xtrain.shape[1]//3], # For regression.
                  'verbose' : [50],
                  'random_state' : [0] # For reproducability of results.
                 }

    custom_cv = custom_cv_2folds(ntrain=Xtrain.shape[0],
                                 nvalid=Xvalid.shape[0])

    qrf_grid = GridSearchCV(qrf,
                            param_grid=parameters,
                            cv=custom_cv,
                            n_jobs=-1)

    qrf_grid.fit(Xtrainvalid, ytrainvalid)
    qrf_best = qrf_grid.best_estimator_ # We save the best estimator.`

where `custom_cv` is the following cross-validation split:
`def custom_cv_2folds(ntrain,nvalid): 
    # Indices for the training and validation partitions are yield.
    idx_train = np.arange(0,ntrain,dtype=int)
    idx_valid = np.arange(ntrain,ntrain+nvalid,dtype=int)
    yield idx_train, idx_valid`

In addition, I think there is no need to provide the data used since there are other issues similar to this one and they do not appear to be data-related.

As the issue has been going on for a while, how it appears to me has changed slightly, thus I will summarize its behaviour in each stage its evolution:

1. At the beginning, I was just running the provided code. The cell in the Jupyter notebook I was executing **hanged forever**. Shutting down the kernel and restarting it manually was unuseful since the task administrator show a 100% usage of the CPU after doing so and the only way to bring my computer back to normality was by rebooting it. In addition to that, sometimes the piece of code provided above might run in case the Grid was smaller. An example of a `GridSearchCV` that **does run successfully**:
`parameters = {'n_estimators' : list(range(10,40,10)),
              'criterion': ['mae'],
              'min_samples_split': list(range(2,11,4)),
              'max_features' : [Xtrain.shape[1]//3], 
              'verbose' : [50],
              'random_state' : [0] 
             }`
I think **`n_estimators` hyper-parameter plays a great role in this**.

2. In this second stage, I was reading about the issue and happened to come across the [10533](https://github.com/scikit-learn/scikit-learn/issues/10533). In which it was recommended to install `cloud pickle` and add the following to my code:
`%env LOKY_PICKLER='cloudpickle' 
import multiprocessing
multiprocessing.set_start_method('forkserver')` which I did.
However, as **I am not using Linux (Windows instead)**, there is no way for me to use that method and set the `forkserver`.
What I did instead, was trying to run my code again, and turned out that now, that the verbose output provided `Using LokyBackend` and the following message showed up:
**`FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details: ValueError: buffer source array is read-only`**

I started looking into it and tried different stuff using [parallel_backend](https://scikit-learn.org/stable/modules/generated/sklearn.utils.parallel_backend.html)

Now, my code runs with `with parallel_backend('threading')`. However, the executing of the trees happened to be unsorted unlike was happening in situations in which the `GridSearchCV` was successfully executed. For instance:
`[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
building tree 1 of 30building tree 1 of 40
building tree 2 of 30
building tree 2 of 40
building tree 3 of 30
building tree 3 of 40
building tree 4 of 30
building tree 4 of 40
building tree 5 of 40
building tree 5 of 30
building tree 1 of 30building tree 6 of 40
building tree 6 of 30building tree 2 of 30
building tree 7 of 40`

I cannot say if the `fit` has finished successfully since the training process takes quite a lot, but I will add that information as soon as it finishes.

**My questions now are**:
1. Is there a reason why this is happening?
2. Is there a workaround for Windows users?
3. I am rather concerned about the disordered executing of the trees during the training process. Do you know why is this happening? Do you think will it be unbeneficial for the model results?

Thanks a lot in advance, and sorry for the long post, but I added all information which I considered relevant so that the investigation had no lack of information.

In case you need me to provide any additional details, just let me know.

Óscar
user: kunfuster
issue_17541: state: open
title: [MRG] Add class_weight parameter to CalibratedClassifierCV
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
To the best of my knowledge, no issues/PRs are addressing this added feature. 


#### What does this implement/fix? Explain your changes.
This PR adds the `class_weight` parameter to the CalibratedClassifierCV class to supplement the already existing `sample_weight` parameter in its `fit` method.
Passing `class_weight='balanced'` to CalibratedClassifierCV will balance the dataset on which the calibration method (sigmoid or isotonic) will be trained (if no `sample_weight` is passed).

#### Any other comments?
Please find attached 4 scripts that test this added parameter : 2 for binary classifications with `cv=prefit` and `cv=2` and 2 for multiclass classifications with `cv=prefit` and `cv=2`.

[class_weighted_calibratedClassifierCV.zip](https://github.com/scikit-learn/scikit-learn/files/4751322/class_weighted_calibratedClassifierCV.zip)


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: amascia
issue_17441: state: open
title: Run common checks on estimators with non default parameters
body: This adds a mechanism for running common checks on estimators with non default parameters. It works as follows,
 1. try to auto-determine valid values for parameters of each estimator (e.g. solvers, losses etc) (https://github.com/scikit-learn/scikit-learn/issues/14063). Currently parameters taken into account are, 
    - categorical parameters:  `"solver",  "algorithm",  "loss",  "strategy", "selection", "criterion", "multi_class", "kernel"`
    - boolean parameter:  `"fit_prior", "fit_intercept", "positive" "normalize", "dual", "average", "shuffle"`.
  The determination for valid parameters for an estimator is bit of a hack but it works in most cases (with a few hard-coded special cases),
   ```py
   >>> from sklearn.tests.test_common_non_default import detect_all_params
   >>> from sklearn.linear_model import LogisticRegression
   >>> detect_all_params(LogisticRegression)
   {'solver': ['lbfgs', 'liblinear', 'newton-cg', 'sag', 'saga'],
    'multi_class': ['auto', 'multinomial', 'ovr'],
    'fit_intercept': [False, True],
    'dual': [False, True]}
   ```
The list of detected parameters can be obtained with
```
python sklearn/tests/test_common_non_default.py
```

<details>

```
ARDRegression
{'fit_intercept': [False, True], 'normalize': [False, True]}
AdaBoostClassifier
{'algorithm': ['SAMME', 'SAMME.R']}
AdaBoostRegressor
{'loss': ['exponential', 'linear', 'square']}
BayesianRidge
{'fit_intercept': [False, True], 'normalize': [False, True]}
BernoulliNB
{'fit_prior': [False, True]}
CategoricalNB
{'fit_prior': [False, True]}
ComplementNB
{'fit_prior': [False, True]}
DecisionTreeClassifier
{'criterion': ['gini', 'entropy']}
DecisionTreeRegressor
{'criterion': ['mse', 'friedman_mse', 'mae']}
DummyClassifier
{'strategy': ['constant', 'most_frequent', 'prior', 'stratified', 'uniform']}
DummyRegressor
{'strategy': ['constant', 'mean', 'median', 'quantile']}
ElasticNet
{'fit_intercept': [False, True],
 'normalize': [False, True],
 'positive': [False, True],
 'selection': ['cyclic', 'random']}
ElasticNetCV
{'fit_intercept': [False, True],
 'normalize': [False, True],
 'positive': [False, True],
 'selection': ['cyclic', 'random']}
ExtraTreeClassifier
{'criterion': ['gini', 'entropy']}
ExtraTreeRegressor
{'criterion': ['mse', 'friedman_mse', 'mae']}
ExtraTreesClassifier
{'criterion': ['gini', 'entropy']}
ExtraTreesRegressor
{'criterion': ['mse', 'friedman_mse', 'mae']}
GammaRegressor
{'fit_intercept': [False, True]}
GaussianProcessClassifier
{'multi_class': ['one_vs_one', 'one_vs_rest']}
GradientBoostingClassifier
{'criterion': ['gini', 'entropy'], 'loss': ['deviance', 'exponential']}
GradientBoostingRegressor
{'criterion': ['mse', 'friedman_mse', 'mae'],
 'loss': ['huber', 'lad', 'ls', 'quantile']}
HistGradientBoostingClassifier
{'loss': ['auto', 'binary_crossentropy', 'categorical_crossentropy']}
HistGradientBoostingRegressor
{'loss': ['least_absolute_deviation', 'least_squares', 'poisson']}
HuberRegressor
{'fit_intercept': [False, True]}
KNeighborsClassifier
{'algorithm': ['auto', 'ball_tree', 'brute', 'kd_tree']}
KNeighborsRegressor
{'algorithm': ['auto', 'ball_tree', 'brute', 'kd_tree']}
KernelRidge
{'kernel': ['precomputed']}
LabelPropagation
{'kernel': ['knn', 'rbf']}
LabelSpreading
{'kernel': ['knn', 'rbf']}
Lars
{'fit_intercept': [False, True], 'normalize': [False, True]}
LarsCV
{'fit_intercept': [False, True], 'normalize': [False, True]}
Lasso
{'fit_intercept': [False, True],
 'normalize': [False, True],
 'positive': [False, True],
 'selection': ['cyclic', 'random']}
LassoCV
{'fit_intercept': [False, True],
 'normalize': [False, True],
 'positive': [False, True],
 'selection': ['cyclic', 'random']}
LassoLars
{'fit_intercept': [False, True],
 'normalize': [False, True],
 'positive': [False, True]}
LassoLarsCV
{'fit_intercept': [False, True],
 'normalize': [False, True],
 'positive': [False, True]}
LassoLarsIC
{'criterion': ['aic', 'bic'],
 'fit_intercept': [False, True],
 'normalize': [False, True],
 'positive': [False, True]}
LinearDiscriminantAnalysis
{'solver': ['eigen', 'lsqr', 'svd']}
LinearRegression
{'fit_intercept': [False, True], 'normalize': [False, True]}
LinearSVC
{'dual': [False, True],
 'fit_intercept': [False, True],
 'loss': ['epsilon_insensitive', 'squared_epsilon_insensitive'],
 'multi_class': ['crammer_singer']}
LinearSVR
{'dual': [False, True],
 'fit_intercept': [False, True],
 'loss': ['epsilon_insensitive', 'squared_epsilon_insensitive']}
LogisticRegression
{'dual': [False, True],
 'fit_intercept': [False, True],
 'multi_class': ['auto', 'multinomial', 'ovr'],
 'solver': ['lbfgs', 'liblinear', 'newton-cg', 'sag', 'saga']}
LogisticRegressionCV
{'dual': [False, True],
 'fit_intercept': [False, True],
 'multi_class': ['auto', 'multinomial', 'ovr'],
 'solver': ['lbfgs', 'liblinear', 'newton-cg', 'sag', 'saga']}
MLPClassifier
{'shuffle': [False, True], 'solver': ['adam', 'lbfgs', 'sgd']}
MLPRegressor
{'shuffle': [False, True], 'solver': ['adam', 'lbfgs', 'sgd']}
MultiTaskElasticNet
{'fit_intercept': [False, True],
 'normalize': [False, True],
 'selection': ['cyclic', 'random']}
MultiTaskElasticNetCV
{'fit_intercept': [False, True],
 'normalize': [False, True],
 'selection': ['cyclic', 'random']}
MultiTaskLasso
{'fit_intercept': [False, True],
 'normalize': [False, True],
 'selection': ['cyclic', 'random']}
MultiTaskLassoCV
{'fit_intercept': [False, True],
 'normalize': [False, True],
 'selection': ['cyclic', 'random']}
MultinomialNB
{'fit_prior': [False, True]}
NuSVC
{'kernel': ['linear', 'poly', 'precomputed', 'rbf', 'sigmoid']}
NuSVR
{'kernel': ['linear', 'poly', 'precomputed', 'rbf', 'sigmoid']}
OrthogonalMatchingPursuit
{'fit_intercept': [False, True], 'normalize': [False, True]}
OrthogonalMatchingPursuitCV
{'fit_intercept': [False, True], 'normalize': [False, True]}
PLSCanonical
{'algorithm': ['nipals', 'svd']}
PassiveAggressiveClassifier
{'average': [False, True],
 'fit_intercept': [False, True],
 'loss': ['hinge', 'squared_hinge'],
 'shuffle': [False, True]}
PassiveAggressiveRegressor
{'average': [False, True],
 'fit_intercept': [False, True],
 'loss': ['epsilon_insensitive', 'squared_epsilon_insensitive'],
 'shuffle': [False, True]}
Perceptron
{'fit_intercept': [False, True], 'shuffle': [False, True]}
PoissonRegressor
{'fit_intercept': [False, True]}
RANSACRegressor
{'loss': ['absolute_loss', 'squared_loss']}
RadiusNeighborsClassifier
{'algorithm': ['auto', 'ball_tree', 'brute', 'kd_tree']}
RadiusNeighborsRegressor
{'algorithm': ['auto', 'ball_tree', 'brute', 'kd_tree']}
RandomForestClassifier
{'criterion': ['gini', 'entropy']}
RandomForestRegressor
{'criterion': ['mse', 'friedman_mse', 'mae']}
Ridge
{'fit_intercept': [False, True],
 'normalize': [False, True],
 'solver': ['auto', 'cholesky', 'lsqr', 'sag', 'saga', 'sparse_cg', 'svd']}
RidgeCV
{'fit_intercept': [False, True], 'normalize': [False, True]}
RidgeClassifier
{'fit_intercept': [False, True],
 'normalize': [False, True],
 'solver': ['auto', 'cholesky', 'lsqr', 'sag', 'saga', 'sparse_cg', 'svd']}
RidgeClassifierCV
{'fit_intercept': [False, True], 'normalize': [False, True]}
SGDClassifier
{'average': [False, True],
 'fit_intercept': [False, True],
 'loss': ['hinge', 'log', 'modified_huber', 'squared_hinge', 'perceptron'],
 'shuffle': [False, True]}
SGDRegressor
{'average': [False, True],
 'fit_intercept': [False, True],
 'loss': ['squared_loss',
          'huber',
          'epsilon_insensitive',
          'squared_epsilon_insensitive'],
 'shuffle': [False, True]}
SVC
{'kernel': ['linear', 'poly', 'precomputed', 'rbf', 'sigmoid']}
SVR
{'kernel': ['linear', 'poly', 'precomputed', 'rbf', 'sigmoid']}
TheilSenRegressor
{'fit_intercept': [False, True]}
TweedieRegressor
{'fit_intercept': [False, True]}
```
</details>
This step can likely be improved further, but it's a start.

2. Compute the Cartesian product of all valid parameters for each estimator.
3. Run estimator checks on the obtained estimators with non default parameters.

This generates 26424 additional tests (the scikit-learn test suite has 16052 tests) which runs in 11 min. The run time can likely be improved by using pytest-xdist on multi-core CPUs .

I think passing these checks would both help detecting new bugs and make our common checks more robust.

This is a somewhat orthogonal solution to https://github.com/scikit-learn/scikit-learn/pull/11324 although they both address the same problem.

There are 2858 errors / 26424. I haven't checked the errors in detail and some might be due to parameter detection failure (in particular some parameters might be incompatible). The logs (without tracebacks) can be found in [pytest.log](https://github.com/scikit-learn/scikit-learn/files/4726732/pytest.log).


cc @NicolasHug @thomasjpfan @jeremiedbb @jnothman 

**TODO:**
 - [ ] run these tests conditionally in CI (maybe in some Cron job)
 - [x] support other types of estimators besides to classifiers and regressors
user: rth
issue_17439: state: open
title: API Proposal for Logging
body: By default writing to a logger would not go to STDOUT without a handler. One solution to this is to attach a handler when verbose is an integer and use the logger if it is passed in.

```py
# Do not log
func(verbose=0)

# will create a stream handler to stdout
func(verbose=1)

# will write to the logger passed in
func(verbose=logger)
```

Edit: See also #78.
user: thomasjpfan
labels: Hard: Hard level of difficulty
issue_17436: state: open
title: Custom Criterion for Tree Splitting
body: #### Describe the workflow you want to enable
I want to be able to define a custom criterion for tree splitting when building decision trees / tree ensembles. 
More specifically, it would be great to be able to base this criterion on features besides X & y (i.e. "Z"), and for that I will need the indexes of the samples being considered. 

#### Describe your proposed solution
clf = DecisionTreeClassifier(..., criterion=my_func)
clf.fit(X, y)

def my_func(indexes):
   return [purity score]

#### Describe alternatives you've considered, if relevant
I looked at the source .pyx code, but it's not clear to me how to start modifying it...
If there was a nice explanation somewhere, it would be a good start.

#### Additional context
I want to define a criterion that takes into consideration a confounding variable.
user: ido-tera-group
issue_17419: state: open
title: Option to make sample_weight for metrics behave like TF (scale to mean of 1)
body: ```py
import tensorflow as tf
import numpy as np
from sklearn.metrics import mean_squared_error

mse = tf.keras.losses.MeanSquaredError()
y_true = [[0, 1, 0], [0, 0, 1]]
y_pred = [[0.05, 0.95, 0], [0.1, 0.8, 0.1]]
sample_weight = [0.3, 0.7]

sample_weight_scaled = np.array(sample_weight) / np.mean(sample_weight)

# no sample_weight, losses are equal
tf_mse = np.float32(mse(y_true, y_pred).numpy())
sk_mse = np.float32(mean_squared_error(y_true, y_pred))
print(tf_mse == sk_mse)

# using the same array of sample weights, produces different values
tf_mse = np.float32(mse(y_true, y_pred, sample_weight=tf.constant(sample_weight)).numpy())
sk_mse = np.float32(mean_squared_error(y_true, y_pred, sample_weight=sample_weight))
print(tf_mse == sk_mse)

# after scaling sample weights to a mean of 1, now produces same output
tf_mse = np.float32(mse(y_true, y_pred, sample_weight=tf.constant(sample_weight_scaled)).numpy())
sk_mse = np.float32(mean_squared_error(y_true, y_pred, sample_weight=sample_weight_scaled))
print(tf_mse == sk_mse)
```

This code above demonstrates that sklearn metrics and losses in TF behave differently when it comes to applying sample weight. While this isn't a big issue on it's own, it becomes a problem when dealing with baselines.

Tensorflow's losses scale the weights to a mean of 1 before applying to the per_sample losses. While sklearn does not.

Would be nice to at least have the option to scale to a mean of 1 to replicate TF values.
user: ben-arnao
issue_17411: state: open
title: Allow Logistic Regression to be Used on Continuous Values
body: Many data science tasks in require fitting a logistic regression on continuous data. Examples include business cycle prediction and many other economics tasks, power curves for wind turbines, and population growth. In fact, any problem with a minimum of zero and some upper bound can have a logistic regression fit to it.

Due to this, logistic regression in sklearn should be modified to enable it to be fit on continuous values instead of only being used for classification.  Currently, logistic regression can be performed with scipy by defining a logistic function and fitting it with scipy.optimize.curve_fit.  However, this is not an sklearn estimator and does not integrate as easily with sklearn. 

user: jdprusa
issue_17401: state: open
title: LinearDiscriminantAnalysis cannot predict_proba if labels are constant
body: I know that most of the times labels will not be constant, however I am using `check_estimator` to check a custom estimator which has a `base_estimator` attribute and scikit-learn chooses LinearDiscriminantAnalysis as base estimator.

#### Describe the bug

```python
import numpy as np
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
rnd = np.random.RandomState(0)
X_train = rnd.uniform(size=(10, 3))
y = np.ones(10)
LinearDiscriminantAnalysis().fit(X_train, y).predict_proba(X_train)
```

The code above will throw the following error:

`AxisError: axis 1 is out of bounds for array of dimension 1`

It does not happen if I use the `predict` method. I also tried with a pair more classifiers (ensembles) and it actually works. I also tried the other discriminant analysis, the quadratic, and it does not let me fit because y is constant. This error makes sense to me, but I can't understand what is going on with the one in the linear discriminant analysis,.

#### Versions
Linux-4.4.0-179-generic-x86_64-with-glibc2.10
Python 3.8.3 (default, May 19 2020, 18:47:26) 
[GCC 7.3.0]
NumPy 1.18.1
SciPy 1.4.1
Scikit-Learn 0.23.1


user: GuillemGSubies
issue_17378: state: open
title: [WIP] Add information gain ratio criterion to tree classifiers
body: #### Reference Issues/PRs

Closes #6821.

#### What does this implement/fix? Explain your changes.

This PR implements the information gain ratio criterion in decision tree classifiers according to the reference:

> J. R. Quinlan, "Induction of Decision Trees", Machine Learning, vol. 1, pp. 81-106, 1986.

which generally leads to considerably smaller decision trees, while obtaining similar scores to other criteria.
user: alfaro96
issue_17377: state: open
title: [WIP] BUG make _weighted_percentile(data, ones, 50) consistent with numpy.median(data)
body: closes #17370 
closes  #6189

Add a new parameter to select the type of interpolation. By default, the median in NumPy and our implementation should be the same.
user: glemaitre
issue_17370: state: open
title: _weighted_percentile does not lead to the same result than np.median
body: While reviewing a test in https://github.com/scikit-learn/scikit-learn/pull/16937, it appears that our implementation of `_weighted_percentile` with unit `sample_weight` will lead to a different result than `np.median` which is a bit problematic for consistency.

In the gradient-boosting, it brakes the loss equivalence because the initial predictions are different. We could bypass this issue by always computing the median using `_weighted_percentile` there.

```python
import pytest
import numpy as np
from sklearn.utils.stats import _weighted_percentile

rng = np.random.RandomState(42)
X = rng.randn(10)
X.sort()
sample_weight = np.ones(X.shape)

median_numpy = np.median(X)
median_numpy_percentile = np.percentile(X, 50)
median_sklearn = _weighted_percentile(X, sample_weight, percentile=50.0)

assert median_numpy == pytest.approx(np.mean(X[[4, 5]]))
assert median_sklearn == pytest.approx(X[4])
assert median_numpy == median_numpy_percentile
assert median_sklearn == pytest.approx(median_numpy)
```
user: glemaitre
issue_17364: state: open
title: PyData/Sparse support
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
Replacing `scipy.sparse` arrays with PyData/Sparse in scikit-learn.

#### Describe your proposed solution
Hello. I'm the maintainer of PyData/Sparse. So far, I feel the conversation between us and the scikit-learn developers has been mostly second-hand.

I'm opening this issue in hopes of opening up a channel, and creating a meta-issue over in the PyData/Sparse repo for any requirements that scikit-learn may have to bring this to fruition. Over time, my hope is we can resolve issues on both sides to enable this.
user: hameerabbasi
issue_17334: state: open
title: KMeans(init='k-means++') performance issue with OpenBLAS
body: I open this issue to investigate a performance problem that might be related to #17230.

I adapted the reproducer of #17230 to display more info and make it work on a medium-size ranom dataset.

```python
from sklearn import cluster
from time import time
from pprint import pprint
from threadpoolctl import threadpool_info
import numpy as np


pprint(threadpool_info())
rng = np.random.RandomState(0)
data = rng.randn(5000, 50)
t0_global = time()
for k in range(1, 15):
    t0 = time()
    # print(f"Running k-means with k={k}: ", end="", flush=True)
    cluster.KMeans(
        n_clusters=k,
        random_state=42,
        n_init=10,
        max_iter=2000,
        algorithm='full',
        init='k-means++').fit(data)
    # print(f"{time() - t0:.3f} s")

print(f"Total duration: {time() - t0_global:.3f} s")
```

I tried to run this on Linux with scikit-learn master (therefore including the #16499 fix)  with 2 different builds of scipy (with openblas from pypi and MKL from anaconda) and various values for `OMP_NUM_THREADS` (unset, `OMP_NUM_THREADS=1`, `OMP_NUM_THREADS=2`, `OMP_NUM_THREADS=4`) on a laptop with 2 physical cpu cores (4 logical cpus).

In both cases, I use the same scikit-learn binaries (built with GCC in editable mode). I just change the env.

The summary is:

- with MKL there is not problem: large or unset values of `OMP_NUM_THREADS` are faster than `OMP_NUM_THREADS=1`;
- with OpenBLAS without explicit setting of `OMP_NUM_THREADS` or setting a large value for it is significanlty slower forced sequential run with `OMP_NUM_THREADS=1`.

I will include my runs in the first comment. 

/cc @jeremiedbb 
user: ogrisel
issue_17322: state: open
title: add method to AgglomerativeClustering which returns the linkage matrix 
body: #### Describe the workflow you want to enable
Many people want to visualize the output of AgglomerativeClustering with a dendrogram. A recent update added an example how to do this (thanks, very useful in teaching!), but people still have to create the linkage matrix which is needed as input to the dendrogram-method in scipy by hand. It would be really cool to be able to plug-in the output of AgglomerativeClustering into dendrogram.

#### Describe your proposed solution
add a method which returns the linkage matrix 

#### Describe alternatives you've considered, if relevant
An attribute like 'linkage_matrix' would be more in line with the workings of scikit-learn, but then, I guess, it would be calculated every time, even if people don't need the matrix. With large numbers of data points the computation time will increase and add to the runtime. So a methods sounds like a better solution


user: fotisj
issue_17321: state: open
title: sklearn.utils.resample stratify parameter is not working
body: I am using 

```
Python 3.6.8
scikit-learn==0.23.1
```

I am using ```sklearn.utils,resample``` for stratified sampling. Here is my code:-

```python
from sklearn.utils import resample
y=[1,1,2,3,2,2,1,3,1,1,2,3,2,2,1,3,4,4]
sample = resample(y, n_samples=5, replace=False, stratify=y,
         random_state=0)
print(sample)
```

gives me output:-
```
[3, 2, 2, 1, 1]
```

Expected output:-
output should contain all the distinct values in y. At least one "4" should be present in the output. For eg.
```
[3,2,2,1,4]
```

Am I missing something? Thank you for your help in advance.
user: tuhinsharma121
issue_17307: state: open
title: Feature weighting for decision trees and random forests
body: **Basic idea:** Allow decision trees and random forests (and possibly other ensemble algorithms) to have feature weightings that make the selection of specific features as split points more or less likely.

https://www.researchgate.net/publication/220338672_Random_feature_weights_for_decision_tree_ensemble_construction

I was looking to see if anything similar existed and came across the above paper.  If I'm understanding the method correctly, the idea is to use random feature weights to create highly diverse ensembles that outperform other common methods, particularly where noise is present.

A general `feature_weights` parameter could allow for tuning based on domain knowledge or other methods that might vastly improve the ability of tree or forest to generalize.  Random forests could accept a set of weights to be used on all forests or an array of weights, one for each tree estimator.

In full disclosure, I'm really just getting my feet wet in terms of decision trees, random forests, etc.  Feel free to let me know if this is wrong or does not make sense.

<!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->


user: jhogg11
issue_17299: state: open
title: Cython and memviews creation
body: Not an issue, just a Cython-related PSA that we need to keep in mind when reviewing PRs:

We shouldn't create 1d views for each sample, this is slow:

```py
cdef float X[:, :] = ...  # big 2d view
for i in range(n_samples):  # same with prange, same with or without the GIL
	f(X[i])
```

do this instead, or use pointers, at least for now:

```py
for i in range(n_samples):
	f(X, i)  # and work on X[i, :] in f's code
```
This is valid for any pattern that generates lots of views so looping over features might not be a good idea either if we expect lots of features.
There might be a "fix" in https://github.com/cython/cython/issues/2227 / cython/cython#3617

The reason is that there's a significant overhead when creating all these 1d views, which comes from Cython internal ref-counting (details at https://github.com/cython/cython/issues/2987). In the hist-GBDT prediction code, this overhead amounts for more than 30% of the runtime so it's not negligible.

Note that:
- Doing this with `prange` used to generate some additional Python interactions, but this was fixed in https://github.com/cython/cython/commit/794d21d929a60c0ff9f1aa92fc79cc79c1d4753d and backported to Cython 0.29
- Now that no Python interactions are generated, we need to be extra careful with this because we won't even see it in Cython annotated files

CC @scikit-learn/core-devs 
user: NicolasHug
issue_17294: state: open
title: LabelEncoder with mixed typed labels feature or bug?
body: Is the following a bug or a feature?

```py
from sklearn.preprocessing import LabelEncoder

LabelEncoder().fit_transform(['1', '2', 2])
# array([0, 1, 1])

LabelEncoder().fit_transform(np.array(['1', '2', 2]))
# array([0, 1, 1])

LabelEncoder().fit_transform(np.array(['1', '2', 2], dtype=object))
# TypeError: Encoders require their input to be uniformly strings or numbers.
```

This stems from how we only check for multiple types in `_encode` when `dtype=object`. Note this behavior is the same in `0.22`, `0.23`, and on master.
user: thomasjpfan
issue_17292: state: open
title: More detailed instructions needed for making (non-English) stop word lists compatible 
body: This issue relates to #10735 re: the improvement of stop word lists.

As stated in the previous discussion, a more detailed documentation for making custom stop-word lists compatible would be more helpful than an updated in-built stop-word list as use cases even within the English language can be very specific.

One of my corpora is a collection of Hiberno-English letters using many outdated word forms as well as words derived from Irish-Gaelic. The txt-files also contain occassional UTF-8 errors and orphaned XML tags inherited from the original documents.

While I acknowledge that removing "lb" for unresolved line-break tags may best be done in the pre-processing, I am also having issues with word forms such as "we'll", "won't" or "'tis" which use apostrophes. Besides, appreviated words such as "oct" for "October" seem to be problematic. 

I have changed my stopword list multiple times to include both word forms with and without apostrophes, but although both "October and "Oct" are on my list, "oct" is still being ignored.

Here is a sample script that might help to update the documentation:

```
from sklearn.feature_extraction.text import CountVectorizer
from sklearn import decomposition
from sklearn.feature_extraction.text import TfidfVectorizer

docs=["Tirconaill tír st sráid Oct 30th 22 A Chait A chara Dhílis 20th I you your me mine faith faithful faithfully ye get got we'll I'll 'tis le length legacy tá tú bhí sé sí &amp amp am"]                                
my_stopwords=["'tis","tis","a", "amp", "30th","Oct","22","ye","I","you","20th","me","get","st", "sráid", "we'll", "ll", "le", "tú", "bhí", "faithfully", "tír"]

print(CountVectorizer(stop_words=my_stopwords).fit_transform(docs).A)

vectorizer=text.CountVectorizer(input='docs', stop_words=my_stopwords) 

dtm=vectorizer.fit_transform(docs).toarray() 

vocab=np.array(vectorizer.get_feature_names())

print(dtm.shape)
print(vocab)
print(len(vocab))

num_topics=2

num_top_words=15

clf=decomposition.NMF(n_components=num_topics, random_state=1)

doctopic=clf.fit_transform(dtm) 

topic_words=[] 
for topic in clf.components_:
   
    word_idx=np.argsort(topic)[::-1][0:num_top_words]
    topic_words.append([vocab[i] for i in word_idx])
    
print(topic_words) 
```

The output I got was this:

>>>[[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]]
>>>(1, 17)
>>>['am' 'chait' 'chara' 'dhílis' 'faith' 'faithful' 'got' 'legacy' 'length' 'mine' 'oct' 'sé' 'sí' 'tirconaill' 'tá' 'we' 'your']
>>>17
>>>[['chait', 'mine', 'sé', 'tá', 'got', 'sí', 'chara', 'length', 'we', 'tirconaill', 'your', 'am', 'legacy', 'faithful', 'oct'], ['faith', 'dhílis', 'oct', 'faithful', 'legacy', 'am', 'your', 'tirconaill', 'we', 'length', 'chara', 'sí', 'got', 'tá', 'sé']]

>>>C:\ProgramData\Anaconda3\lib\site-packages\sklearn\feature_extraction\text.py:300: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['oct', 'we'] not in stop_words.
  'stop_words.' % sorted(inconsistent))

user: MonikaBarget
issue_17288: state: open
title: Multilabel KNeighborsClassifier out-of-memory 
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
<!--
A clear and concise description of what the bug is.
-->
The `KNeighborsClassifier` crashes on `fit` or `predict` with `X` very large in both dimensions (many samples, many classes) in the multilabel setting, because the sparse indicator array `y` is densified in an intermediate step.

If fixing this is worthwhile, I can create a PR. Please let me know.

#### Steps/Code to Reproduce
<!--
Please add a minimal example that we can reproduce the error by running the
code. Be as succinct as possible, do not depend on external data. In short, we
are going to copy-paste your code and we expect to get the same
result as you.
-->

Example:
```python
from sklearn.datasets import make_multilabel_classification
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

X, y = make_multilabel_classification(n_samples=1_001_000,
                                      n_features=2,
                                      n_classes=100_000,
                                      random_state=123,
                                      sparse=False,
                                      return_indicator='sparse',
                                     )

# MemoryError in fit on large training data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=1_000, shuffle=False)
print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)
knn = KNeighborsClassifier(n_neighbors=5, n_jobs=4)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
```
and
```python
# MemoryError in predict on large test data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=1_000_000, shuffle=False)
print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)
knn = KNeighborsClassifier(n_neighbors=5, n_jobs=4)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
```

#### Expected Results
No error is thrown. I assume there is no true requirement to densify the complete `y` at any point.

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
```
MemoryError                               Traceback (most recent call last)
<ipython-input-39-3eba0dff4be0> in <module>
     16 print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)
     17 knn = KNeighborsClassifier(n_neighbors=5, n_jobs=4)
---> 18 knn.fit(X_train, y_train)
     19 y_pred = knn.predict(X_test)

~/miniconda3/envs/accelerator/lib/python3.7/site-packages/sklearn/neighbors/_base.py in fit(self, X, y)
   1144         check_classification_targets(y)
   1145         self.classes_ = []
-> 1146         self._y = np.empty(y.shape, dtype=np.int)
   1147         for k in range(self._y.shape[1]):
   1148             classes, self._y[:, k] = np.unique(y[:, k], return_inverse=True)

MemoryError: Unable to allocate 745. GiB for an array with shape (1000000, 100000) and data type int64
```
and
``` 
MemoryError                               Traceback (most recent call last)
<ipython-input-38-1124c82d6e9f> in <module>
      4 knn = KNeighborsClassifier(n_neighbors=5, n_jobs=4)
      5 knn.fit(X_train, y_train)
----> 6 y_pred = knn.predict(X_test)

~/miniconda3/envs/accelerator/lib/python3.7/site-packages/sklearn/neighbors/_classification.py in predict(self, X)
    182         weights = _get_weights(neigh_dist, self.weights)
    183 
--> 184         y_pred = np.empty((n_queries, n_outputs), dtype=classes_[0].dtype)
    185         for k, classes_k in enumerate(classes_):
    186             if weights is None:

MemoryError: Unable to allocate 745. GiB for an array with shape (1000000, 100000) and data type object
```
#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
import imblearn; print("Imbalanced-Learn", imblearn.__version__)
-->
```
System:
    python: 3.7.5 (default, Oct 25 2019, 15:51:11)  [GCC 7.3.0]
executable: /PATH/TO/miniconda3/envs/accelerator/bin/python
   machine: Linux-3.10.0-1127.el7.x86_64-x86_64-with-centos-7.8.2003-Core

Python dependencies:
       pip: 20.0.2
setuptools: 46.0.0.post20200309
   sklearn: 0.22.1
     numpy: 1.18.1
     scipy: 1.4.1
    Cython: 0.29.15
    pandas: 1.0.2
matplotlib: 3.1.3
    joblib: 0.14.1

Built with OpenMP: True
```
(offending lines still there in latest sklearn master, 0.23)


<!-- Thanks for contributing! -->

user: VarIr
issue_17286: state: open
title: the use of chi2 with the iris dataset is incorrect 
body: #### Describe the issue linked to the documentation:
many of the examples in the documentation uses the iris data set, and some also use the chi2 for feature selection. however the since the iris data is continuous the use of chi2 as a metric is invalid. 

here are the examples i found with this error:

https://scikit-learn.org/stable/auto_examples/svm/plot_svm_anova.html#sphx-glr-auto-examples-svm-plot-svm-anova-py

https://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection

#### Suggest a potential alternative/fix
change the dataset or the metric. or bin the data.

user: OfirKedem
issue_17234: state: open
title: RAM footprint of IterativeImputer grows with each iteration, causing MemoryError
body: **Description**
I was trying to emulate missForest by pairing IterativeImputer with RandomForestRegressor (same with ExtraTreesRegressor). Calling fit_transform() causes a MemoryError on my machine (16GB RAM) after some iterations. Tested on 0.23 and 0.22.2.post1

```
Python dependencies:
       pip: 20.1
setuptools: 46.3.0.post20200513
   sklearn: 0.23.0
     numpy: 1.18.4
     scipy: 1.3.1
    Cython: None
    pandas: 1.0.3
matplotlib: 3.2.1
    joblib: 0.14.1
Built with OpenMP: True
```

**Code used**
```
import pandas
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
from sklearn.ensemble import RandomForestRegressor

X = pandas.read_csv("DATA" + ".csv")  #  matrix with shape (273944, 5)
mf = IterativeImputer(RandomForestRegressor(random_state=0, n_jobs=-1), random_state=0, verbose=2)
X_ = mf.fit_transform(X)
```


**Solution (local)**
Commenting out line 647 "self.imputation_sequence_.append(estimator_triplet)" in _iterative.py solved the problem for me, leading to constant RAM usage. This attribute is used in transform(), but not reused in fit_transform().

user: danuko42
issue_17212: state: open
title: "Paired" distance function for haversine distance
body: #### Describe the workflow you want to enable
I want to be able to calculate paired distance between 2 arrays with equal dimension, using haversine distance.

#### Describe your proposed solution
https://stackoverflow.com/questions/29545704/fast-haversine-approximation-python-pandas/29546836#29546836

#### Describe alternatives you've considered, if relevant
I can roll my own function using the above code from stackoverflow, but I'd rather import from sklearn.

#### Additional context

Currently [paired_distances](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/pairwise.py#L912-L918) supports the following options:

```python
PAIRED_DISTANCES = {
    'cosine': paired_cosine_distances,
    'euclidean': paired_euclidean_distances,
    'l2': paired_euclidean_distances,
    'l1': paired_manhattan_distances,
    'manhattan': paired_manhattan_distances,
    'cityblock': paired_manhattan_distances}
```

There's code out there on stackoverlow for a fast and paired implementation of haversine distance: https://stackoverflow.com/questions/29545704/fast-haversine-approximation-python-pandas/29546836#29546836

It seems like this would be useful to have in sklearn
user: zachmayer
issue_17195: state: open
title: get idf value of words from TfidfVectorizer like a dictionary
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
Currently, if we want to get idf value from TfidfVectorizer, we can use
`    from sklearn.feature_extraction.text import TfidfVectorizer

    my_data = ["hello how are you", "hello who are you", "i am not you"]

    tf = TfidfVectorizer(use_idf=True)
    tf.fit_transform(my_data)

    idf = tf.idf_ `

but it cannot show the idf value for 'hello', like what we can do with term frequency in TfidfVectorizer.vocabulary_['hello']. 

#### Describe your proposed solution
Maybe the group can add a dictionary object in TfidfVectorizer, so we can have TfidfVectorizer.idf_dict.get()
#### Describe alternatives you've considered, if relevant

#### Additional context
And honestly, I think the current **vocabulary_** in TfidfVectorizer is misleading. The user would think it would get the tf-idf value, but it actually gets the frequency.
user: dujiaxin
issue_17184: state: open
title: sklearn.tree.plot_tree - Adding path labels to decision tree plots
body: Hi,

Would it be possible to add "True"/"False" labels (or any other adequate naming) to the paths from the root node to first-level leaves when splitting? For instance, when instantiating a model with DecisionTreeClassifer(max_depth=1) and visualizing it via plot_tree, there is no option to add path labels to the plot.

This would be really helpful for those reasons:
- To make visually clear, which path is True which one is False, just by looking into the plots, for binary classifications
- For multiclass classification, the labels would be really helpful to visually understand which path represents that particular class in a decision tree

Would it be possible to implement such feature as a parameter to plot_tree() in the future in sklearn?

Kind regards,
DK
user: d-kleine
issue_17181: state: open
title: Lasso.path() same as ElasticNet.path()?
body: The [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html#sklearn.linear_model.Lasso.path) for the `path()` method of `Lasso()` suggests that it is the same as `ElasticNet.path()` (see [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html#sklearn.linear_model.ElasticNet.path)) with `l1_ratio=0.5`. This is, of course, no longer the same Lasso regression as in the documentation for `Lasso()`. Should the defaults for this `path()` method be changed such that by default it computes the Lasso path (i.e. `l1_ratio=1` or something similar)?

Also, I have a general question on best practices when using `path()` functionality (for my own cross-validation methods). In the [Glmnet Vignette](https://web.stanford.edu/~hastie/Papers/Glmnet_Vignette.pdf), at the very end of the document, it is suggested that the entire regularisation path should be fitted, then specific lambdas (alphas in sklearn terminology) inputted.

> Alternatively, a more stable and strongly recommended way to perform this task is in two stages. First fit the entire lasso or elastic-net path without specifying lambda, letting it chose its own sequence. Then make a call to coef or predict (using the exact = TRUE option) and provide the requested λ0 to extract the corresponding coefficients. (If λ0 is not in the λ sequence generated by glmnet, the path will be refitted along a new λ sequence that augments the original sequence with the new value(s)).

In my case, I am looking to extract the coefficients for a sequence of lambdas, so I use a list of them for the `alphas` argument to `path()`. However, I am not sure if this is most efficient. It is certainly more efficient / faster than running `Lasso.set_params(alpha=new_alpha)` for each `new_alpha` in the list.

Finally, what is the advantage of using `lasso_path()` versus `Lasso().path(l1_ratio=1)`?

user: sachin-smart
issue_17180: state: open
title: ExtraTreeRegressor doctest fails on precision check on aarch64 platforms
body: 
#### Describe the bug

The `ExtraTreeRegressor` doctest fails on a precision check on aarch64 platforms. When running the same tests on x86 CI, the tests pass successfully.

#### Steps/Code to Reproduce

Please see https://gitlab.com/libreml/libreml/pipelines/144575194

On x86 CI (https://gitlab.com/libreml/libreml/-/jobs/545952560/artifacts/file/cache/buildstream/logs/libreml/scikit-learn/dedc3c59-build.10059.log)

```bash
14304 passed, 156 skipped, 1 deselected, 6 xfailed, 1621 warnings in 870.10s (0:14:30)
```

On aarch64 CI (https://gitlab.com/libreml/libreml/-/jobs/545952561/artifacts/file/cache/buildstream/logs/libreml/scikit-learn/2d0895af-build.31041.log)

```bash
         ______________ [doctest] sklearn.tree._classes.ExtraTreeRegressor ______________
        1639     >>> from sklearn.model_selection import train_test_split
        1640     >>> from sklearn.ensemble import BaggingRegressor
        1641     >>> from sklearn.tree import ExtraTreeRegressor
        1642     >>> X, y = load_boston(return_X_y=True)
        1643     >>> X_train, X_test, y_train, y_test = train_test_split(
        1644     ...     X, y, random_state=0)
        1645     >>> extra_tree = ExtraTreeRegressor(random_state=0)
        1646     >>> reg = BaggingRegressor(extra_tree, random_state=0).fit(
        1647     ...     X_train, y_train)
        1648     >>> reg.score(X_test, y_test)
        Expected:
            0.7823...
        Got:
            0.7825994290971925
        /buildstream/libreml/scikit-learn.bst/sklearn/tree/_classes.py:1648: DocTestFailure
```

```bash
         = 1 failed, 14303 passed, 156 skipped, 1 deselected, 6 xfailed, 1622 warnings in 1894.16s (0:31:34) =
        [00:44:56] FAILURE scikit-learn.bst: Running build-commands
        [00:45:42] FAILURE scikit-learn.bst: Command 'pytest -m "not network" $(echo "import sklearn; print(sklearn.__path__[0])" | python3 -u)' failed with exitcode 1
```

#### Versions

0.22.2.post1

user: cphang99
labels: arch:arm: ARM related issues
module:test-suite: everything related to our tests
issue_17179: state: open
title: [WIP] FIX KBinsDiscretizer: allow nans #9341
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
This fixes #9341

#### What does this implement/fix? Explain your changes.
This fix extends the functionality of the KBinsDiscretizer to allow for NaNs in the dataset. The NaN will be assigned to an additional category -1 and this category will be propagated into the onehot-encoding if specified.

#### Any other comments?
The new feature in KBinsDiscretizer did not pass the test
```
pytest sklearn/tests/test_common.py -k KBinsDiscretizer
```
getting the following assertion error:
```
AssertionError: ("Estimator doesn't check for NaN and inf in fit.", KBinsDiscretizer())
```
Since it does not need to check for NaN anymore I'm just wondering if this test is needed. In the case it is not needed anymore, then the pull request is complete :-)

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: PabloRMira
issue_17171: state: open
title: How to get eigenvectors of kernel pca
body: Hello,

I'm using kernel pca to reduce dimensionality and I need eigenvalues and eigenvectors. In PCA, I know pca.explained_variance_ is eigenvalues and pca.components_ is eigenvectors. I read the sklearn document and found the below words in kpca.
```
lambdas_ : array, (n_components,)
Eigenvalues of the centered kernel matrix in decreasing order.

alphas_ : array, (n_samples, n_components)
Eigenvectors of the centered kernel matrix.
```
Compare with pca's document, I'm confused about why the eigenvectors's shape is not equal. In pca, the shape is (n_components, n_features) while kpca is (n_samples, n_components). Here is pca's document.
```
explained_variance_array, shape (n_components,)
The amount of variance explained by each of the selected components.Equal to n_components largest eigenvalues of the covariance matrix of X.

components_ : array, shape (n_components, n_features)
Principal axes in feature space, representing the directions of maximum variance in the data. The components are sorted by explained_variance_.
```
I know if the kpca's kernel is `linear`, it is exactly pca. So I want know how to get eigenvalues and eigenvectors in kpca. Can you help me?
user: s0rel
issue_17170: state: open
title: Extend plot_precision_recall_curve and plot_roc_curve to multiclass
body: Similarly to the example provided here: https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html, extend plot_precision_recall_curve and plot_roc_curve to be able to deal with multiclass scenarios.

user: claramatos
issue_17168: state: open
title: [RFC] Registry for scorers
body: `sklearn.metrics.SCORERS` acts like registry for getting predefined scorers in `cross_val_score`, `cross_validate`, `*SearchCV` estimators, etc. If someone wants to use his/her custom scoring functions in the aforementioned functions/classes should define the scoring function, make it scorer using the `make_scorer` function and pass it around. Another option could be to patch the `sklearn.metrics.SCORERS`.

Instead of patching directly the `SCORERS` the `make_scorer` could optionally register the returned scorer. Another, and probably better, solution could be to create a new decorator or to refactor the `make_scorer` to act as a decorator (it can actually) that accepts arguments. Like this, one could just define a scoring function and it will be available everywhere in `scikit-learn` as string. Contrib developers could leverage this functionality as well.

What do you think?
user: chkoar
issue_17158: state: open
title: set_params() changing the result of transform / predict without fitting
body: I came across this thing

```py
from sklearn.feature_selection import SelectFromModel
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression

X, y = make_classification(n_samples=1000, random_state=0)

est = SelectFromModel(LogisticRegression(), threshold=.4)
est.fit(X, y)
print(est.transform(X).shape)  # 1000, 3
est.threshold = .01
print(est.transform(X).shape)  # 1000, 20
```

where the output of transform was changed by just setting an `__init__` parameter.

Is this a bug or a feature? If it's a bug, it's easy to fix. If it's not, then I  have a few concerns:

- This typically isn't possible for any parameter than needs to be validated in `fit`, so we can't support this pattern  consistently
- We don't do that in e.g. PCA where it would be in theory possible, and quite useful. I.e. users might want to transform to different values of n_components without having to refit all the time.
user: NicolasHug
issue_17140: state: open
title: impute.KNNImputer always uses just the mean value
body: #### Describe the bug
Trying to use KNNImputer with a Pandas dataframe to impute missing values. It always just uses the mean value of the column instead of actually trying to impute smoother values.

#### Steps/Code to Reproduce
```python
import pandas as pd
from sklearn.impute import KNNImputer

df = pd.read_csv('pydf.csv', index_col=0)
imputer = KNNImputer()
dfnew = df.copy(deep=True)
exguess = imputer.fit_transform(dfnew[['Incoming Examinations']])
exguess
```

This is pydf.csv:
```
,Incoming Examinations,Year,Month,Date
19,362.0,2006,1,2006-01-01
10,436.0,2006,2,2006-02-01
57,,2006,3,2006-03-01
23,490.0,2006,4,2006-04-01
12,508.0,2006,5,2006-05-01
70,,2006,6,2006-06-01
68,393.0,2006,7,2006-07-01
63,596.0,2006,8,2006-08-01
36,634.0,2006,9,2006-09-01
29,613.0,2006,10,2006-10-01
46,545.0,2006,11,2006-11-01
50,411.0,2006,12,2006-12-01
72,398.0,2007,1,2007-01-01
43,311.0,2007,2,2007-02-01
91,664.0,2007,3,2007-03-01
92,680.0,2007,4,2007-04-01
87,507.0,2007,5,2007-05-01
11,467.0,2007,6,2007-06-01
77,566.0,2007,7,2007-07-01
76,806.0,2007,8,2007-08-01
95,732.0,2007,9,2007-09-01
40,886.0,2007,10,2007-10-01
64,776.0,2007,11,2007-11-01
0,698.0,2007,12,2007-12-01
88,875.0,2008,1,2008-01-01
26,840.0,2008,2,2008-02-01
5,724.0,2008,3,2008-03-01
53,1115.0,2008,4,2008-04-01
90,997.0,2008,5,2008-05-01
35,775.0,2008,6,2008-06-01
75,886.0,2008,7,2008-07-01
49,1041.0,2008,8,2008-08-01
59,1011.0,2008,9,2008-09-01
86,,2008,10,2008-10-01
45,939.0,2008,11,2008-11-01
7,,2008,12,2008-12-01
28,1004.0,2009,1,2009-01-01
60,1065.0,2009,2,2009-02-01
22,1263.0,2009,3,2009-03-01
4,962.0,2009,4,2009-04-01
52,,2009,5,2009-05-01
81,1429.0,2009,6,2009-06-01
94,1205.0,2009,7,2009-07-01
24,890.0,2009,8,2009-08-01
38,1320.0,2009,9,2009-09-01
37,1276.0,2009,10,2009-10-01
84,1757.0,2009,11,2009-11-01
58,1710.0,2009,12,2009-12-01
54,1710.0,2010,1,2010-01-01
34,1709.0,2010,2,2010-02-01
67,1578.0,2010,3,2010-03-01
30,1604.0,2010,4,2010-04-01
74,1758.0,2010,5,2010-05-01
1,,2010,6,2010-06-01
71,1457.0,2010,7,2010-07-01
85,1607.0,2010,8,2010-08-01
16,1808.0,2010,9,2010-09-01
65,1866.0,2010,10,2010-10-01
2,1934.0,2010,11,2010-11-01
3,2294.0,2010,12,2010-12-01
44,,2011,1,2011-01-01
27,2334.0,2011,2,2011-02-01
41,1973.0,2011,3,2011-03-01
61,2262.0,2011,4,2011-04-01
82,2259.0,2011,5,2011-05-01
15,2217.0,2011,6,2011-06-01
42,2739.0,2011,7,2011-07-01
89,2772.0,2011,8,2011-08-01
8,3383.0,2011,9,2011-09-01
14,2869.0,2011,10,2011-10-01
48,2239.0,2011,11,2011-11-01
33,,2011,12,2011-12-01
78,2789.0,2012,1,2012-01-01
39,3455.0,2012,2,2012-02-01
17,2940.0,2012,3,2012-03-01
79,2968.0,2012,4,2012-04-01
25,3466.0,2012,5,2012-05-01
73,3037.0,2012,6,2012-06-01
47,3946.0,2012,7,2012-07-01
62,3459.0,2012,8,2012-08-01
20,3446.0,2012,9,2012-09-01
51,3258.0,2012,10,2012-10-01
55,4729.0,2012,11,2012-11-01
21,3694.0,2012,12,2012-12-01
9,4610.0,2013,1,2013-01-01
32,4841.0,2013,2,2013-02-01
80,5172.0,2013,3,2013-03-01
93,4351.0,2013,4,2013-04-01
18,4730.0,2013,5,2013-05-01
31,4706.0,2013,6,2013-06-01
66,5000.0,2013,7,2013-07-01
6,4978.0,2013,8,2013-08-01
56,5008.0,2013,9,2013-09-01
83,6094.0,2013,10,2013-10-01
13,4874.0,2013,11,2013-11-01
69,5806.0,2013,12,2013-12-01
```

#### Actual Results
This is exguess:
```
array([[ 362.        ],
       [ 436.        ],
       [2060.13636364],
       [ 490.        ],
       [ 508.        ],
       [2060.13636364],
       [ 393.        ],
       [ 596.        ],
       [ 634.        ],
       [ 613.        ],
       [ 545.        ],
       [ 411.        ],
       [ 398.        ],
       [ 311.        ],
       [ 664.        ],
       [ 680.        ],
       [ 507.        ],
       [ 467.        ],
       [ 566.        ],
       [ 806.        ],
       [ 732.        ],
       [ 886.        ],
       [ 776.        ],
       [ 698.        ],
       [ 875.        ],
       [ 840.        ],
       [ 724.        ],
       [1115.        ],
       [ 997.        ],
       [ 775.        ],
       [ 886.        ],
       [1041.        ],
       [1011.        ],
       [2060.13636364],
       [ 939.        ],
       [2060.13636364],
       [1004.        ],
       [1065.        ],
       [1263.        ],
       [ 962.        ],
       [2060.13636364],
       [1429.        ],
       [1205.        ],
       [ 890.        ],
       [1320.        ],
       [1276.        ],
       [1757.        ],
       [1710.        ],
       [1710.        ],
       [1709.        ],
       [1578.        ],
       [1604.        ],
       [1758.        ],
       [2060.13636364],
       [1457.        ],
       [1607.        ],
       [1808.        ],
       [1866.        ],
       [1934.        ],
       [2294.        ],
       [2060.13636364],
       [2334.        ],
       [1973.        ],
       [2262.        ],
       [2259.        ],
       [2217.        ],
       [2739.        ],
       [2772.        ],
       [3383.        ],
       [2869.        ],
       [2239.        ],
       [2060.13636364],
       [2789.        ],
       [3455.        ],
       [2940.        ],
       [2968.        ],
       [3466.        ],
       [3037.        ],
       [3946.        ],
       [3459.        ],
       [3446.        ],
       [3258.        ],
       [4729.        ],
       [3694.        ],
       [4610.        ],
       [4841.        ],
       [5172.        ],
       [4351.        ],
       [4730.        ],
       [4706.        ],
       [5000.        ],
       [4978.        ],
       [5008.        ],
       [6094.        ],
       [4874.        ],
       [5806.        ]])
```

2060.13636364 is the mean value of that column. I expected something much closer to its neighbors.

#### Versions
```
>>> import sklearn; sklearn.show_versions()

System:
    python: 3.7.7 (default, Apr 15 2020, 05:09:04) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\flori\Anaconda3\python.exe
   machine: Windows-10-10.0.18362-SP0

Python dependencies:
       pip: 20.0.2
setuptools: 46.1.3.post20200330
   sklearn: 0.22.1
     numpy: 1.18.1
     scipy: 1.4.1
    Cython: 0.29.17
    pandas: 1.0.3
matplotlib: 3.1.3
    joblib: 0.14.1

Built with OpenMP: True
```

Running in a Jupyter notebook on Windows, all installed with Anaconda.
user: FlorinAndrei
issue_17138: state: open
title: no k-modes or k-prototypes algorithms in scikit-learn
body: I noticed that k-means is specially useful for numerical sample, while for categorical data sets, it may use algorithms like k-modes or k-prototypes. May i know why scikit does not include these kinds of algorithms for categorical data as these data is very common in reality, so wondering any background or any plan to support it
 
Thanks
user: mayujsw
issue_17137: state: open
title: Add sample table for Error-Correcting Output-Codes
body: I felt that the explanation around Error-Correcting Output-Codes could have been improved by adding an example / table.

#### Reference Issues/PRs
Improvable explanation of how Error-Correcting Output-Codes works and how the codes are estimated via sklearn.

#### What does this implement/fix? Explain your changes.
Adds a table and an example.
user: joeDiHare
issue_17135: state: open
title: One Class SVM default nu value 0 (pure dataset)
body: I wonder why mathematically a pure dataset can not be explicitely fed to one-class svm.
We can put nu to very low value (nu=0.01). But still some of the "correct"/normal data will be tagged as outliers.
Even if the method of Schokolpd itself does not provide I believe such an explanation would greatly benefit and and enhance already existing parts of the documentation 
(parts that I find relevant
https://scikit-learn.org/stable/modules/outlier_detection.html#outlier-detection

https://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html#sklearn.svm.OneClassSVM.decision_function )

I have read the original paper also but I fail to understand the reason at least for now.
(https://papers.nips.cc/paper/1723-support-vector-method-for-novelty-detection.pdf)

scikit often fills the gap between "deep-level" interepreration but I believe in this case the documentation would definitely benefit from. 

If I solve the question should I make a pull request ?  
Best regards and thank you for the great job maintaining this library.







user: partizanos
issue_17130: state: open
title: Gaussian Mixture - weighted implementation 
body: PR created to work on  #15647

Implementation of a weighted GMM version. If a sample has a weight w, it means the sample is observed w times.

Based on the two papers:
- EM Algorithms for Weighted-Data Clustering with Application to Audio-Visual Scene Analysis, Gebru et al., https://arxiv.org/abs/1509.01509
- Reward-weighted GMM and its application toaction-selection in robotized shoe dressing, Colomé et al., http://www.iri.upc.edu/files/scidoc/1953-Reward-weighted-GMM-and-its-application-to-action-selection-in-robotized-shoe-dressing.pdf

Replicating the structure of Kmeans with a sample_weight argument.
Two main tests are added:
- samples with zero weights do not have an impact
- samples with integer weights should give the same result than a duplicated sample (ex: if a sample X1 has a weight of 3 and all other samples have a unit weight, weighted gmm should give the same result that the classical GMM with three X1 samples + the other samples with no modification)

For the moment, the work is still in progress, in particular regarding:
- PEP8
- the validity of the EM algorithm
- the tests
- the default behavior of sample_weight (if None)

user: dupuisIRT
issue_17075: state: open
title: cross_validate could have an option to return any object based on target and predictions 
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
The `cross_validate` method is able to return multiple scoring metrics, which makes it very flexible. However, this option does not allow for returning arrays (or even a custom object), which is interesting if you wanted to draw a cross-validated ROC curve, or a cross-validated calibration curve. Currently, you would have create your custom `cross_validate` method to return these type of metrics.

#### Describe your proposed solution
I think there could be an option similar to the `scoring` kwarg (maybe 'func'?), that accepts a dictionary with the name of the desired output and the function (ROC curve, for instance). The function should have the same signature as required by the `make_scorer` method (i.e., the function should accept the target, and the predicted classes or probabilities), except by the flexibility in the returned object. 

Indeed, this option would return the output of the function in the same way the current API returns the trained models for each CV fold.
user: franzoni315
issue_17069: state: open
title: Enable stop/resume in GridSearchCV and RandomizedSearchCV
body: #### Describe the workflow you want to enable
I would like it to be possible to stop and resume GridSearchCV and RandomizedSearchCV. This is relevant in situations where training a model can take a long time.

Right now, as far as I know, if the execution stops for whatever reason the results are lost.

#### Describe your proposed solution
I propose an extra parameter in GridSearchCV and RandomizedSearchCV: `cache_results_folder` or something more appropriate that behaves as follows:

- After a set of cross-validation run (5-fold, say) is completed, a file is saved in `cache_results_folder` containing the set of parameters for that run
- In the case of GridSearchCV, if a log exists for that combination of parameters, load results and skip the run for that combination
- In the case of RandomizedSearchCV, when logs exist just increase the `n_iter` count

#### Describe alternatives you've considered, if relevant
It may be viable to also save the current CV split in the log, but I assume that would increase the effort considerably.
I considered manually running smaller CV calls and saving intermediate results but that seems suboptimal.

#### Additional context
Nowadays training neural networks as classifiers is very common. However, it is not unusual to get memory leaks or invalid parameter combinations when doing hyperparameter search (randomized, for example). This feature would bring more robustness to GridSearchCV and RandomizedSearchCV.
user: douglas125
issue_17058: state: open
title: GridsearchCV.score with multimetric scoring and callable refit
body: #### Describe the bug
When using GridsearchCV with multimetric scoring and a callable as refit, the `GridsearchCV.score` function doesn't works since `score = self.scorer_[self.refit] if self.multimetric_ else self.scorer_` seems to wait only for a string in case of multimetric scoring
#### Steps/Code to Reproduce
```
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, RepeatedStratifiedKFold
from sklearn.linear_model import LogisticRegression
import numpy as np

def get_best_index(cv_results):
    best_rank_mask = cv_results['rank_test_roc_auc'] == cv_results['rank_test_roc_auc'].min()
    params_best_score = np.array(cv_results['params'])[best_rank_mask]
    params_name = params_best_score[0].keys()
    classifier_params_names = [_ for _ in params_name if 'classifier' in _]
    if 'classifier__C' in classifier_params_names or 'classifier__base_estimator__C' in classifier_params_names:
        classifier_params = [_['classifier__C' if 'classifier__C' in classifier_params_names else 'classifier__base_estimator__C'] for _ in params_best_score]
        params_best_score = params_best_score[classifier_params == min(classifier_params)]
    best_params = params_best_score[0]
    best_index = int(np.where(np.array(cv_results['params']) == best_params)[0])
    return best_index

breast = load_breast_cancer()
X = breast.data
y = breast.target
cv = RepeatedStratifiedKFold(5, 2, random_state=111)
params_dic = {'C': np.arange(0.1, 1.1, 0.1)}
clf = GridSearchCV(LogisticRegression(penalty='l2', max_iter=1e5, solver='saga'), params_dic, scoring=['roc_auc', 'accuracy'], cv=cv, refit=get_best_index, n_jobs=4)
clf.fit(X,y)
clf.score(X,y)
```

#### Actual Results
File "C:\Users\Tim\Anaconda3\lib\site-packages\sklearn\model_selection\_search.py", line 447, in score
    score = self.scorer_[self.refit] if self.multimetric_ else self.scorer_
KeyError: <function get_best_index at 0x0000028DD66BDBF8>

#### Expected Results
Since refit is a callable I don't know how he could know which metric to choose for scoring., However, if I give a string with the metric I chose, i.e. 'roc_auc', to refit argument the best index won't be chosen in the way I want. Maybe in case of multimetric scoring and callable refit, ask for dictionnary instead like {score: callable} and the score will be used in GridsearchCV.score ?

#### Versions
System:
    python: 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]
executable: C:\Users\Tim\Anaconda3\python.exe
   machine: Windows-10-10.0.18362-SP0
Python dependencies:
       pip: 20.0.2
setuptools: 39.1.0
   sklearn: 0.22.1
     numpy: 1.18.1
     scipy: 1.4.1
    Cython: 0.28.2
    pandas: 1.0.0
matplotlib: 2.2.2
    joblib: 0.14.1
Built with OpenMP: True

user: TimZaragori
issue_17052: state: open
title: ENH Adds permute_y to make_classification interaction
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes https://github.com/scikit-learn/scikit-learn/issues/16789


#### What does this implement/fix? Explain your changes.
Permutes the labels instead of generating new labels.

#### Any other comments?
Not backward compatible. 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
issue_17048: state: open
title: New Feature: return_all_estimators option in GridSearch and RandomizedSearch
body: #### New feature: return_all_estimators option in GridSearch and RandomizedSearch
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Implements new feature as requested in #16765. 

#### What does this implement/fix? Explain your changes.
New option is included to the search functions in order to refit all the estimators and return them.
Here's a list of the changes:
- Any `BaseSearchCV` object now includes an init parameter `return_all_estimators` (set to False by default).
- If flag is set to True then all the refitted estimators are stored in a list under the variable `<BaseSearchCV object>.all_estimators_`
- The estimators are refitted and stored only if `refit=True`
- The `refit_time` will include the time needed to fit the rest of the models if `return_all_estimators=True`

#### Any other comments?
Please review the changes and if you agree with the overall solution and then I'll move forward with updating the documentation.

user: chrispe
issue_17034: state: open
title: all gradient calculated are scaled by length_scale parameter in Matern kernel
body: Hi, there's a subtle bug in the numerical gradient calculation for Matern covariance kernels. The num. gradient is used when there is no simple closed form gradient. 

So the issue is that the function that is fed into ``_approx_fprime`` essentially uses ``kernel.theta + dx`` as the test point. But ``kernel.theta`` is the log of the hyperparameters. So consider varying one parameter p: instead of doing

(f(p + delta) - f(p)) / delta

it's calculating

(f(p + delta * p) - f(p)) / delta

Aside from other unintentional consequences, the current calculation is off by a factor of p.

1. ``np.exp(theta)`` should be given to ``_approx_fprime``
2. ``np.log(theta)`` should be used in the helper function def'd just before this ref

Sorry for being in a bit of a rush. Hope it's pretty straightforward, though.

https://github.com/scikit-learn/scikit-learn/blob/2955d9f0af7785ed353d6ad23740f62852fcf988/sklearn/gaussian_process/kernels.py#L1638
user: miketrumpis
issue_16994: state: open
title: [MRG] ENH Support column vector input to feature extraction
body: Resolves #15588
Resolves #16972

To Do:

* [x] CountVectorizer, TfidfVectorizer
   * [x] tests
   * [x] docs
* [x] DictVectorizer
   * [x] tests
   * [x] docs
* ~~HashingVectorizer (tricky if input_type='string')~~ postponed. See https://github.com/scikit-learn/scikit-learn/pull/16994#issuecomment-618190827
   * tests
   * docs
user: jnothman
issue_16990: state: open
title: Getter/Setter for fit parameters
body: #### Describe the workflow you want to enable
Robust cross-version persistence and serialization of sklearn models.  
Motivation: I like reproducibility and portability :) 

#### Describe your proposed solution

`BaseEstimator.get_params()` and `set_params()` is great for getting the initialization state of models and a step towards full deserialization. I've already cracked serializing arbitrary nested data structs (comprising usual JSON primitives, plus datetimes, numpy arrays, and a few other custom data types) to JSON. What I need is a way of knowing which attributes comprise the fitted parameters (I shall refer to them as `state_params`, this is different than `fit_params`, I believe those are just `**kwargs` which get passed to `estimator.fit()`). If model classes implement a small API specifying which parameters are necessary, it's pretty easy to write any sort of serializer for the model (json, protobuf, whatever).

I can achieve similar functionality by mutating `model.__dict__` but that doesn't provide any contract between scikit versions. 

Something akin to
```python
class SerializableMixin:
    """
    All estimators should specify all the parameters necessary to 
capture the full state of the model. 
To implementing this mixin, populate _get_state_param_names, and the getter/setter methods
    """

    @classmethod
    @abstractmethod
    def _get_state_param_names(cls):
        """Get name of state parameters (all params besides those returned by
        _get_param_names) necessary to recreate model state"""
        # Could probably store these as a class_level _state_param_names 

    def get_state_params(self, deep=True):
        """Get parameters for this estimator.
        Return a mapping of state parameters and their values
        """
        out = dict()
        for key in self._get_state_param_names():
            value = getattr(self, key, None)
            if deep and hasattr(value, 'get_state_params'):
                deep_items = value.get_params().items()
                out.update((key + '__' + k, val) for k, val in deep_items)
            out[key] = value
        return out

    def set_state_params(self, **params):
        """Set the state parameters of this estimator.
This is an all-or-nothing affair, you must provide every parameter that the model 
needs to de/serialize or it'll throw an error
        """
        some_check_that_target_supports_state_params()
        some_check_that_all_state_params_are_provided()
        if not params:
            # Simple optimization to gain speed (inspect is slow)
            return self
        valid_params = self.get_params(deep=True)
        for key, value in params.items():
            ...
        return self
```

#### Describe alternatives you've considered, if relevant
pickle, which throws the warning that inter-version pickled models may not be compatible. 
I searched through the issues for related terms and haven't found this specific approach mentioned. 

user: xkortex
issue_16988: state: open
title: Support numpy.random.Generator and/or BitGenerator for random number generation
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->


#### Describe the workflow you want to enable

I'd like to use a `Generator` or `BitGenerator` with scikit-learn where I'd otherwise use `RandomState` or a `seed` int.

For example:
```
import numpy as np

bit_generator = np.random.PCG64(seed=0)
generator = np.random.Generator(bit_generator)
```
and then use this for `random_state=` in scikit-learn:
```
from sklearn.datasets import make_classification
from sklearn.model_selection import ShuffleSplit
from sklearn.svm import LinearSVC

X, y = make_classification(random_state=generator)  # or my bit_generator here 
classifier = LinearSVC(random_state=generator)
cv = ShuffleSplit(random_state=generator)
```

This fails because these methods expect a `RandomState` object or `int` seed value. The specific trigger is `check_random_state(random_state)`. 

#### Describe your proposed solution

This would require:
- changing code to allow `Generator` or `BitGenerator` as acceptable values for `random_state=..` in every function and class constructor that accepts `random_state`.
- change `check_random_state()` to allow `Generator` and/or `BitGenerator` objects.
- adding tests for using `Generator` or `BitGenerator` with classes or functions that consume `random_state` (similar to `seed` int or `RandomState` objects already)
- change any internal code that assumes `RandomState` methods that aren't available with `Generator` (e.g. `rand`, `randn`, see )
- maybe switch to using `Generator` instead of `RandomState` by default, when seed int is given

#### Describe alternatives you've considered, if relevant

The scope could include either or both of `BitGenerator` or `Generator`.

It might be easiest to allow only `BitGenerator`, and not `Generator`.
- This allows flexibility. 
  - Users have control over seed and PRNG algorithm.
- This is easier to implement (can be treated just like a `seed` int value).
  - `BitGenerator` can be given to `RandomState`, and I think it then produces the same values as `Generator`.

#### Additional context

NumPy v1.17 added the `numpy.random.Generator` ([docs](https://numpy.org/doc/1.18/reference/random/generator.html)) interface for random number generation. 

Overview:
- `Generator` is similar to `RandomState`, but enables different PRNG algorithms
- `BitGenerator` ([docs](https://numpy.org/doc/1.18/reference/random/bit_generators/index.html)) encapsulates the PRNG and seed value, e.g. [`PCG64(seed=0)`](https://numpy.org/doc/1.18/reference/random/bit_generators/pcg64.html) 
- `RandomState` "is considered frozen" and uses "the slow Mersenne Twister" by default ([docs](https://numpy.org/doc/1.17/reference/random/legacy.html))
- `RandomState` can work with non-Mersenne `BitGenerator` objects
- More info in [NEP-19](https://numpy.org/neps/nep-0019-rng-policy.html), the design document from NumPy.

The API for `Generator` and `BitGenerator` looks like:

```
from numpy import random

bit_generator = random.PCG64(seed=0)  # PCG64 is a BitGenerator subclass
generator = random.Generator(bit_generator)

generator.uniform(...)  # API is similar to RandomState

# there's also this, for making a PCG64-backed Generator
generator = random.default_rng(seed=0)
```

user: grisaitis
issue_16986: state: open
title: Initialization of tSNE with PCA, allow for 'auto' instead of 'random'
body: **#### Describe the workflow you want to enable**

The current sklearn initialization of TSNE is `'random'` by default. The other option is to initialize it with `'pca'`. However, when you set `init='pca'`, it uses the `svd_solver='randomized'` by default. What happens is if you rerun the algorithm it will keep generating different results. Currently the only way to avoid it is to set `random_state` to some value. Unless your dataset is larger than 500 features, there is no need to work with `'randomized'` svd since it doesn't produce consistent results and is only used for computational speed.


**#### Describe your proposed solution**

The current code for TSNE initialization in sklearn is the following:
```
if isinstance(self.init, np.ndarray):
    X_embedded = self.init
elif self.init == 'pca':
     pca = PCA(n_components=self.n_components, svd_solver='randomized',
               random_state=random_state)
    X_embedded = pca.fit_transform(X).astype(np.float32, copy=False)
elif self.init == 'random':
    # The embedding is initialized with iid samples from Gaussians with
    # standard deviation 1e-4.
    X_embedded = 1e-4 * random_state.randn(
        n_samples, self.n_components).astype(np.float32)
else:
    raise ValueError("'init' must be 'pca', 'random', or a numpy array")
```
My proposed solution is to set `svd_solver='auto'` which will solve the issue and help a lot because in this case it will use a `'full'` svd for  smaller datasets and switch to `'randomized'` for very large datasets.
My proposed code change:
```
if isinstance(self.init, np.ndarray):
    X_embedded = self.init
elif self.init == 'pca':
     pca = PCA(n_components=self.n_components, svd_solver='auto',
               random_state=random_state)
    X_embedded = pca.fit_transform(X).astype(np.float32, copy=False)
elif self.init == 'random':
    # The embedding is initialized with iid samples from Gaussians with
    # standard deviation 1e-4.
    X_embedded = 1e-4 * random_state.randn(
        n_samples, self.n_components).astype(np.float32)
else:
    raise ValueError("'init' must be 'pca', 'random', or a numpy array")
```


#### Describe alternatives you've considered, if relevant
This can be done outside of the function by creating a PCA instance first and then using it to initialize TSNE and by default with 'auto' svd solver:
```
pca = PCA()
tsne = TSNE(init=pca)
```
compared to current way with default 'randomized' svd solver:
`tsne = TSNE(init='pca')`
In this case, to keep results consistent through multiple iteration you would need to set `random_state`, whereas in my proposed solution it will do full svd for most datasets.
user: misha-pyshark
issue_16980: state: open
title: StratifiedKFold doesnt work with bytes classes
body: #### Describe the bug

I'm using this dataset https://www.openml.org/d/1515 where classes (20 in total) are represented as bytes. When trying to run cross-validation I got the error message:

```py
ValueError: You appear to be using a legacy multi-label data representation. Sequence of sequences are no longer supported; use a binary array or sparse matrix instead - the MultiLabelBinarizer transformer can convert to this format.
```

I think this message is not proper, because there is single label for each sample. 

#### Code to Reproduce

```py
import numpy as np
import pandas as pd
from sklearn.model_selection import StratifiedKFold
# generate data
nrows = 100
ncols = 3
X = np.random.rand(nrows,ncols)
y = pd.Series([b'a', b'b']*int(nrows/2))
# show y
print(y)
print(type(y[0]))
# split data
skf = StratifiedKFold(n_splits=2)
for train, test in skf.split(X, y):
    print(train, test)
```
I got the error:
```py
~/sandbox/mljar-supervised/venv/lib/python3.6/site-packages/sklearn/utils/multiclass.py in type_of_target(y)
    258         if (not hasattr(y[0], '__array__') and isinstance(y[0], Sequence)
    259                 and not isinstance(y[0], str)):
--> 260             raise ValueError('You appear to be using a legacy multi-label data'
    261                              ' representation. Sequence of sequences are no'
    262                              ' longer supported; use a binary array or sparse'

ValueError: You appear to be using a legacy multi-label data representation. Sequence of sequences are no longer supported; use a binary array or sparse matrix instead - the MultiLabelBinarizer transformer can convert to this format.
```

When I change the `y` to `str` it works as expected:

```py
y = y.astype(str)
skf = StratifiedKFold(n_splits=2)
for train, test in skf.split(X, y):
    print(train, test)
```
prints:
```py
[50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73
 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97
 98 99] [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
 48 49]
[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
 48 49] [50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73
 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97
 98 99]
```

#### Versions
scikit-learn 0.22.2



user: pplonski
labels: Needs Decision: Requires decision
issue_16976: state: open
title: kmeans elkan vs. lloyd performance 
body: I tried kmeans fit_predict performance on 0.23-dev0 with mnist_all.csv dataset (http://ratml.org/datasets/mnist_all.csv). It seems elkan cost more time than lloyd (no matter dense or sparse format input),  not sure whether this is due to dgemm vectorization. so wondering whether this is as expected, if yes, will we change algorithm parameter 'auto' logic to redirect the algorithem to lloyd ? 

And is there any on-going work or plan to improve elkan perf ? Thanks 
user: mayujsw
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_16972: state: open
title: Allow 2d array input in CountVectorizer?
body: I wonder if we should allow ``(n_samples, 1)`` input to CountVectorizer. I think we had this discussion before, right now we basically have a special case in ``ColumnTransformer`` with scalar columns to produce 1d arrays.
Does anyone remember where this discussion was?

A lot of my students are struggling with putting a CountVectorizer into a ColumnTransformer and it might be nice to be more generous with the input formats, and ensure we raise nice error messages.

```python
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
X = pd.DataFrame(["what is my purpose?", "You bring the butter."])
CountVectorizer().fit_transform(X)
```
```pythontb
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-6-6cae33e8c706> in <module>
      3 from sklearn.compose import ColumnTransformer
      4 X = pd.DataFrame(["what is my purpose?", "You bring the butter."])
----> 5 CountVectorizer().fit_transform(X)

~/miniconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in fit_transform(self, raw_documents, y)
   1218 
   1219         vocabulary, X = self._count_vocab(raw_documents,
-> 1220                                           self.fixed_vocabulary_)
   1221 
   1222         if self.binary:

~/miniconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _count_vocab(self, raw_documents, fixed_vocab)
   1129         for doc in raw_documents:
   1130             feature_counter = {}
-> 1131             for feature in analyze(doc):
   1132                 try:
   1133                     feature_idx = vocabulary[feature]

~/miniconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _analyze(doc, analyzer, tokenizer, ngrams, preprocessor, decoder, stop_words)
    101     else:
    102         if preprocessor is not None:
--> 103             doc = preprocessor(doc)
    104         if tokenizer is not None:
    105             doc = tokenizer(doc)

~/miniconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _preprocess(doc, accent_function, lower)
     66     """
     67     if lower:
---> 68         doc = doc.lower()
     69     if accent_function is not None:
     70         doc = accent_function(doc)

AttributeError: 'int' object has no attribute 'lower'
```


user: amueller
issue_16962: state: open
title: add stratified split for "shuffle=False" in train_test_split
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable

When splitting time series data, data is often split without shuffling. But now `train_test_split` only supports stratified split with `shuffle=True`.
It would be helpful to add stratify option for `shuffle=False` also.

#### Describe your proposed solution

Add option `shuffle` in `StratifiedShuffleSplit`, and only permutate indices when `shuffle` option is `True`.
Then, use `StratifiedShuffleSplit` with shuffle option in `train_test_split`.   
For example:
```
def train_test_split(*arrays, **options):
    ...
    if shuffle is False:
        if stratify is not None:
            cv = StratifiedShuffleSplit(test_size=n_test, train_size=n_train,
                         random_state=random_state, shuffle=False)
            train, test = next(cv.split(X=arrays[0], y=stratify))
        else:
            train = np.arange(n_train)
            test = np.arange(n_train, n_train + n_test)
    ...

class StratifiedShuffleSplit(BaseShuffleSplit):
    def __init__(self, n_splits=10, test_size=None, train_size=None,
                 random_state=None, shuffle=True):
        super().__init__(
            n_splits=n_splits,
            test_size=test_size,
            train_size=train_size,
            random_state=random_state)
        self._default_test_size = 0.1
        self.shuffle=shuffle

    def _iter_indices(self, X, y, groups=None):
        n_samples = _num_samples(X)
        y = check_array(y, ensure_2d=False, dtype=None)
        
        ...

        rng = check_random_state(self.random_state)

        for _ in range(self.n_splits):
            # if there are ties in the class-counts, we want
            # to make sure to break them anew in each iteration
            n_i = _approximate_mode(class_counts, n_train, rng)
            class_counts_remaining = class_counts - n_i
            t_i = _approximate_mode(class_counts_remaining, n_test, rng)

            train = []
            test = []

            for i in range(n_classes):
                if self.shuffle:
                    permutation = rng.permutation(class_counts[i])
                    perm_indices_class_i = class_indices[i].take(permutation,
                                                                 mode='clip')
                else:
                    perm_indices_class_i = class_indices[i]

                train.extend(perm_indices_class_i[:n_i[i]])
                test.extend(perm_indices_class_i[n_i[i]:n_i[i] + t_i[i]])

            if self.shuffle:
                train = rng.permutation(train)
                test = rng.permutation(test)

            yield train, test
```

#### Describe alternatives you've considered, if relevant

Alternative idea is adding split process directly in train_test_split (like case `Shuffle=False` and stratify is None), or add new method. But that new code is very similar to StratifiedShuffleSplit, so modifying StratifiedShuffleSplit would be better.
user: Yukimura66
issue_16956: state: open
title: BUG: segfaults in testsuite with PyPy on ppc64le
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug

While working on https://github.com/conda-forge/scikit-learn-feedstock/pull/123, I found that some tests segfault specifically on ppc64le with PyPy (but not with other runtimes on ppc64le, or with PyPy on other platforms). I gather that there's no CI for PyPy here, but in any case, @rth [asked](https://github.com/conda-forge/scikit-learn-feedstock/pull/123#issuecomment-614625410) me to open a bug here.

Here are the logs from the [CI run](https://travis-ci.com/github/conda-forge/scikit-learn-feedstock/jobs/319979951) .

```
=================================== FAILURES ===================================
________________________ ensemble/tests/test_bagging.py ________________________
[gw0] linux -- Python 3.6.9 $PREFIX/bin/python
worker 'gw0' crashed while running 'ensemble/tests/test_bagging.py::test_sparse_classification'
________________________ linear_model/tests/test_sgd.py ________________________
[gw2] linux -- Python 3.6.9 $PREFIX/bin/python
worker 'gw2' crashed while running 'linear_model/tests/test_sgd.py::test_sgd_averaged_computed_correctly[SGDRegressor]'
________________________ metrics/tests/test_ranking.py _________________________
[gw1] linux -- Python 3.6.9 $PREFIX/bin/python
worker 'gw1' crashed while running 'metrics/tests/test_ranking.py::test_score_scale_invariance'
__________________ manifold/tests/test_spectral_embedding.py ___________________
[gw3] linux -- Python 3.6.9 $PREFIX/bin/python
worker 'gw3' crashed while running 'manifold/tests/test_spectral_embedding.py::test_spectral_embedding_callable_affinity[sparse]'
rker 'gw3' crashed while running 'manifold/tests
=============================== warnings summary ===============================
[...]
```


user: h-vetinari
labels: module:test-suite: everything related to our tests
issue_16955: state: open
title: BUG: Test failure on ppc64le
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug

While working on https://github.com/conda-forge/scikit-learn-feedstock/pull/123, I found that `manifold/tests/test_t_sne.py::test_uniform_grid[barnes_hut]` fails on ppc64le; @rth [asked](https://github.com/conda-forge/scikit-learn-feedstock/pull/123#issuecomment-614625410) me to open a bug here.

```
=========================== short test summary info ============================
FAILED manifold/tests/test_t_sne.py::test_uniform_grid[barnes_hut] - Assertio...
= 1 failed, 13718 passed, 398 skipped, 6 xfailed, 1674 warnings in 382.15s (0:06:22) =
```

Here are the logs, the failure is the same across cpython 3.6-3.8:
* [python 3.6](https://travis-ci.com/github/conda-forge/scikit-learn-feedstock/jobs/319979948)
* [python 3.7](https://travis-ci.com/github/conda-forge/scikit-learn-feedstock/jobs/319979949)
* [python 3.8](https://travis-ci.com/github/conda-forge/scikit-learn-feedstock/jobs/319979950)

Interestingly, [PyPy](https://travis-ci.com/github/conda-forge/scikit-learn-feedstock/jobs/319979951) doesn't have that failure (but has some segfaults instead..., see #16956).

A more detailed trace is below the fold. From what I can see from the log, the failure seems to be a question of "only" the quality of an embedding, and not a hard failure per se.

<details>

```
=================================== FAILURES ===================================
________________________ test_uniform_grid[barnes_hut] _________________________
[gw2] linux -- Python 3.6.10 $PREFIX/bin/python

method = 'barnes_hut'

    @pytest.mark.parametrize('method', ['barnes_hut', 'exact'])
    def test_uniform_grid(method):
        """Make sure that TSNE can approximately recover a uniform 2D grid
    
        Due to ties in distances between point in X_2d_grid, this test is platform
        dependent for ``method='barnes_hut'`` due to numerical imprecision.
    
        Also, t-SNE is not assured to converge to the right solution because bad
        initialization can lead to convergence to bad local minimum (the
        optimization problem is non-convex). To avoid breaking the test too often,
        we re-run t-SNE from the final point when the convergence is not good
        enough.
        """
        seeds = [0, 1, 2]
        n_iter = 500
        for seed in seeds:
            tsne = TSNE(n_components=2, init='random', random_state=seed,
                        perplexity=20, n_iter=n_iter, method=method)
            Y = tsne.fit_transform(X_2d_grid)
    
            try_name = "{}_{}".format(method, seed)
            try:
>               assert_uniform_grid(Y, try_name)

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:784: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

Y = array([[ 52.326397  , -15.92225   ],
       [ 46.679527  , -20.175953  ],
       [ 40.870537  , -24.181147  ],
       ...[-35.291374  ,  22.122814  ],
       [-42.2738    ,  18.793724  ],
       [-48.922283  ,  15.606232  ]], dtype=float32)
try_name = 'barnes_hut_1'

    def assert_uniform_grid(Y, try_name=None):
        # Ensure that the resulting embedding leads to approximately
        # uniformly spaced points: the distance to the closest neighbors
        # should be non-zero and approximately constant.
        nn = NearestNeighbors(n_neighbors=1).fit(Y)
        dist_to_nn = nn.kneighbors(return_distance=True)[0].ravel()
        assert dist_to_nn.min() > 0.1
    
        smallest_to_mean = dist_to_nn.min() / np.mean(dist_to_nn)
        largest_to_mean = dist_to_nn.max() / np.mean(dist_to_nn)
    
        assert smallest_to_mean > .5, try_name
>       assert largest_to_mean < 2, try_name
E       AssertionError: barnes_hut_1
E       assert 6.67359409617653 < 2

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:807: AssertionError

During handling of the above exception, another exception occurred:

method = 'barnes_hut'

    @pytest.mark.parametrize('method', ['barnes_hut', 'exact'])
    def test_uniform_grid(method):
        """Make sure that TSNE can approximately recover a uniform 2D grid
    
        Due to ties in distances between point in X_2d_grid, this test is platform
        dependent for ``method='barnes_hut'`` due to numerical imprecision.
    
        Also, t-SNE is not assured to converge to the right solution because bad
        initialization can lead to convergence to bad local minimum (the
        optimization problem is non-convex). To avoid breaking the test too often,
        we re-run t-SNE from the final point when the convergence is not good
        enough.
        """
        seeds = [0, 1, 2]
        n_iter = 500
        for seed in seeds:
            tsne = TSNE(n_components=2, init='random', random_state=seed,
                        perplexity=20, n_iter=n_iter, method=method)
            Y = tsne.fit_transform(X_2d_grid)
    
            try_name = "{}_{}".format(method, seed)
            try:
                assert_uniform_grid(Y, try_name)
            except AssertionError:
                # If the test fails a first time, re-run with init=Y to see if
                # this was caused by a bad initialization. Note that this will
                # also run an early_exaggeration step.
                try_name += ":rerun"
                tsne.init = Y
                Y = tsne.fit_transform(X_2d_grid)
>               assert_uniform_grid(Y, try_name)

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:792: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

Y = array([[-18.169476  ,   6.0802336 ],
       [-18.278513  ,   2.8822129 ],
       [-18.671782  ,  -0.4646889 ],
       ...[ 22.550077  ,  19.698557  ],
       [ 21.399723  ,  22.933178  ],
       [ 16.22136   ,  28.22955   ]], dtype=float32)
try_name = 'barnes_hut_1:rerun'

    def assert_uniform_grid(Y, try_name=None):
        # Ensure that the resulting embedding leads to approximately
        # uniformly spaced points: the distance to the closest neighbors
        # should be non-zero and approximately constant.
        nn = NearestNeighbors(n_neighbors=1).fit(Y)
        dist_to_nn = nn.kneighbors(return_distance=True)[0].ravel()
        assert dist_to_nn.min() > 0.1
    
        smallest_to_mean = dist_to_nn.min() / np.mean(dist_to_nn)
        largest_to_mean = dist_to_nn.max() / np.mean(dist_to_nn)
    
        assert smallest_to_mean > .5, try_name
>       assert largest_to_mean < 2, try_name
E       AssertionError: barnes_hut_1:rerun
E       assert 2.145051767903112 < 2

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:807: AssertionError
=============================== warnings summary ===============================
[...]
```

</details>

user: h-vetinari
labels: module:test-suite: everything related to our tests
issue_16951: state: open
title: Inconsistent navigation on user guide
body: Not sure when this happened but i love the new left-hand side navigation https://scikit-learn.org/dev/user_guide.html

(@adrinjalali did this maybe?)
![image](https://user-images.githubusercontent.com/449558/79594368-e3a10e00-80aa-11ea-8007-c5628ea84ac9.png)

However, when clicking the different entries, the result is inconsistent. For some, it shows a TOC that expands the existing toc with more detail:
![image](https://user-images.githubusercontent.com/449558/79594421-f9aece80-80aa-11ea-9320-e1e9d0ae3d1e.png)

For others it replaces the TOC entirely:
![image](https://user-images.githubusercontent.com/449558/79594455-08958100-80ab-11ea-8b7b-e3b528495622.png)

I prefer the first one and I think we should adjust the TOC on all of the pages so that the high-level overview stays present on all of the pages.

That requires changes of getting started, glossary, api reference and examples.

user: amueller
issue_16939: state: open
title: score() and score_samples()
body: #### Describe the issue linked to the documentation

<!--
Tell us about the confusion introduced in the documentation.
-->
 Hi all, 

target:I am trying to predict the bandwidth of the KernelDensity() on sklearn by using randomized search and bayesian optimization (fmin() from hyperopt). 

Issue: score_samples() and score() are not really clear to me. In the source code, the probability density is mentioned, then the total log density and likelihood. The are already challenging concepts by themselves and I find them hard to interpret when implementing my code. For instance I get a negative score which for  a log density would imply the log of a very small probability number if I am not wrong, which I find it odd. I have browsed the internet many times  and there are no real examples. 
For instance, understanding a PDF when it comes to determining the chance of rain between a specific interval is relatively easy, but when the model is about other things, like interest rate, bandwidth (using KernelDensity()) the interpretation is not so straightforward. The same goes for the likelihood, one thing is interpreting it when it is about a coin toss, and another thing is when it is about more variables combine together. 

Furthermore, the surrogate function which is defined with a cross_val_score gives a negative score, so also that one is not of immediate interpretation, and cross_val_score takes the default function, i.e. score() of the KernelDensity() when the scoring is not applied. Of course, the choice of the optimal bandwidth should be "somewhat" explained by the best score result. 

Could someone with more expertise than me make these source codes explanations more clear or provide examples on how to interpret the results when applying score_samples() and score() on real stuff and not using 'accuracy' but the default score function on KernelDensity() ? 

I found that np.sum(bestestimator.score_samples(X)) = score() and it is possible to plug also the np.exp to get np.sum(np.exp(bestestimator.score_samples(X))) to get the total log density but i get values of -100 or even lower so it is hard to interpret. 

Thanks for the help (code is prop so i cannot provide it, but i can elaborate, if needed). 

#### Suggest a potential alternative/fix

<!--
Tell us how we could improve the documentation in this regard.
--> Expand the set of example code with non standard scenarios, using kernelDensity with randomized search and on a regression framework (not classification). Also it would be nice if you could clarify/check those definitions of score and score_samples in the source code which are quite confusing ( I am not a statistician). 

thanks 

user: lbegatti
issue_16931: state: open
title: make_regression should include categorical variables with weights
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

Currently `make_regression` does not do anything with categorical variables. You can only simulate continuous variables that can then be converted to categorical variables (e.g., `1 if x>0 else 0`). However in many cases it is necessary to have categorical variables (e.g., a city ID).

In many cases these categorical variables are not equally distributed. There may be many rows in a data set for some category values while few rows for other data. Thus the fraction of categories present should be available as an option like they are for outcome variables in make_classification.

If this is already a feature somewhere and I am unaware, I am happy to contribute to that feature instead.
user: mnky9800n
issue_16920: state: open
title: Random state seed parameter for agglomerative clustering 
body: 
#### Describe the workflow you want to enable
It will be helpful to have to have random state seed in  agglomerative clustering like in Kmeans

class sklearn.cluster.KMeans(n_clusters=8, init='k-means++', n_init=10, max_iter=300, tol=0.0001, precompute_distances='auto', verbose=0, **random_state=None**, copy_x=True, n_jobs=None, algorithm='auto')



#### Describe your proposed solution
Current 
class sklearn.cluster.AgglomerativeClustering(n_clusters=2, affinity='euclidean', memory=None, connectivity=None, compute_full_tree='auto', linkage='ward', distance_threshold=None)

Proposed
class sklearn.cluster.AgglomerativeClustering(n_clusters=2, affinity='euclidean', memory=None, connectivity=None, compute_full_tree='auto', linkage='ward', distance_threshold=None,**random_state=seed**)


user: aliasneo1
issue_16917: state: open
title: Harmonize the "full" (`PCA`) vs "dense" (`KernelPCA`) names for `eigen_solver` parameter
body: Suggestion made by @ogrisel in PR #12069. 

These two names mean the same thing, we should probably deprecate one.
Note that equivalent parameters are probably hidden in other models, this is an opportunity to look for them.


user: smarie
issue_16911: state: open
title: fit_interncept in nonlinear svm
body: Hi, 

In svm.LinearSVC there is a parameter "fit_intercept" which controls the bias and can be turned on or off. Why isn't there such an option for non-linear svm? Could you please add it or guide me towards adding it? 
user: nastaran75
issue_16898: state: open
title: Tutorial on warmstart grid search for multiple classifiers
body: Solves issue #16371

This tutorial tunes hyperparameters max_features and n_estimators for ExtraTrees and RandomForestClassifier in order to fairly compare their performance.

**Included in Functions** 

- Given classifiers and dictionaries of two hyperparameters, finds the optimal pair of hyperparameter values for each classifier using grid search. In this specific example, max_features and n_estimators will be the parameters being tuned.

-  Implements warmstart for searching through increasing numbers of estimators in the ensemble.

- Visualizes the performance of each parameter value pair in a heatmap to show more detail on how the optimal parameters were determined.
user: cshih14
issue_16881: state: open
title: [MRG] OneHotEncoder handle_unknown param can be set to 'warning'
body: **Reference Issues/PRs**

None

**What does this implement/fix? Explain your changes.**

When running pipeline in production and an unknown categorical feature is present during transform, I would like it to be transformed as in handle_unknown='ignore', but I would also like to be notified about it for monitoring reasons. To handle this issue, one would have to wrap the current implementation of OneHotEncoder, catch the specific error and return warning.

This PR is to address the issue by only warning when unknown features are presented. 

**Any other comments?**
user: RoeiKa
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_16880: state: open
title: Confusion Matrix colors do not reflect the classification performance
body: I am not sure whether this really is an issue, but I tried to find a way to resolve this but I had no success so far. I apologize in advance if there is an already implemented easy way to solve this, but I tried searching and nothing came up. I am attaching pictures with the code I used to explain my issue. 

so I am using the following code in order to evaluate prediction/label from two lists:

```python
from sklearn.metrics import classification_report, confusion_matrix, ConfusionMatrixDisplay
import matplotlib.pyplot as plt
    
target_names = ['Empty', 'Human', 'Dog', 'Dog&Human']
labels_names = [0,1,2,3] 
print(classification_report(y_true, y_pred,labels=labels_names, target_names=target_names))    
cm = confusion_matrix(y_true, y_pred,labels=labels_names, normalize='pred')
disp = ConfusionMatrixDisplay(confusion_matrix=cm,display_labels=target_names)
disp = disp.plot(cmap=plt.cm.Blues,values_format='g')
plt.show()
```
When I use the attribute `normalize='pred'`, everything appears as it should be

![image](https://user-images.githubusercontent.com/33216367/78868449-eb92f980-7a7d-11ea-9fc4-55526cb75c7a.png)

However, if I decide that I wanna show the exact number of instances predicted in the Confusion Matrix and remove the normalize attribute, the heatmap does not represent the precision, but rather the number of data. So a class detected with 60% of precision appears darker than one with 90% if the first has more instances of data than the second. 

![image](https://user-images.githubusercontent.com/33216367/78869095-f4d09600-7a7e-11ea-8c05-8893aa5f103d.png)

Is there a way to make the color match the precision and not the number without having to normalize ? 

Thanks a lot

user: Wazaki-Ou
issue_16873: state: open
title: Choose first feature to use in Tree and/or choose the order of features to use in the Tree
body: The idea: Allow a user to pre-select which feature to first split off of and/or allow a user to pre-select which order of the features to split off of.

First example: I might want to first split on a seasonal feature. So, the model would be forced to split on that feature first, and optimize entropy/gini from there. (I am not telling the tree which value to split off of, just which feature to use.)

Second example: I might want to force the model to use 5 features in a specific order. The model would be forced to optimize entropy/gini in the order I specify.

Third example: I might want to force the model to use 3 features in a specific order, but I might pass 10 features to the model. The model would be forced to use the first 3, in the order I specify, then the model could use which other features necessary as specified by the other params.

The story: In finance/trading, sometimes we wish to force which features to make decisions off of (and the order!): all we need is an algo that tells us which levels to trade off of. 

We also find it useful to first force a seasonal difference (e.g., winter vs summer), then build a tree from there (one for winter and one for summer). Yes, we could build two trees after first splitting a dataset on our criteria; however, having one model that would do this makes things more succinct. Furthermore, when splitting the dataset like this, the number of rows gets reduced (bad). Mind you, in finance, sometimes it's difficult to get a lot of data.

One might argue that this defeats the purpose of using a model, like a decision tree, which optimizes the levels to trade and chooses which features are more important, but it doesn't. Sometimes optimizing is too prone to overfitting. Plus, the combination of human knowledge (subject matter experts) and data driven models can be better than one, or the other.

If this isn't the best algo or place to suggest this at, I would love some feedback. I would also love to see someone else enhance this idea.
user: Dylan-Cunningham
issue_16856: state: open
title: Document order of patches in extract_patches_2d
body: #### Describe the issue linked to the documentation

AFAICT, when `max_patches` is None and the input array is C-ordered `feature_extraction.image.extract_patches_2d` returns all patches "in the same order as would naive nested loops iterating in C order" -- IOW, the order expected by `reconstruct_from_patches_2d`.  However this is not documented; in fact the behavior for `max_patches=None` (the default) isn't documented either.

#### Suggest a potential alternative/fix

- Document that `max_patches=None` returns all patches (I can write a PR for that).
- Document order in which the patches are returned, and whether the input array needs to be C-contiguous to achieve that (that I don't know, and I don't know whether you want to guarantee the return order as well).
user: anntzer
issue_16846: state: open
title: In multifold.MDS  stress value doesn’t correspond to returned coordinates
body: <p><strong>Description</strong></p>
<p>&nbsp;</p>
<p>In multifold.MDS&nbsp; stress value doesn&rsquo;t correspond to returned coordinates.</p>
<p>&nbsp;</p>
<p><strong>How to reproduce:</strong></p>
<p>&nbsp;</p>
<p>Apply MDS to four points in 2d-plane:</p>
<p>&nbsp;</p>
<p>X &nbsp;&nbsp;Y</p>
<p>1 &nbsp;&nbsp;5</p>
<p>1 &nbsp;&nbsp;4</p>
<p>1&nbsp; &nbsp;1</p>
<p>3&nbsp; &nbsp;3</p>
<p>&nbsp;</p>
<p>If distances are Euclidean, then disparities matrix will be:</p>
<p>&nbsp;</p>
<p>disparities =</p>
<p>&nbsp;[ 0.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;4.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2.82842712&nbsp; ]</p>
<p>&nbsp;[ 1.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;3.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2.23606798&nbsp; ]</p>
<p>&nbsp;[ 4.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 3.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2.82842712&nbsp; ]</p>
<p>&nbsp;[ 2.82842712&nbsp; &nbsp; &nbsp; 2.23606798&nbsp; &nbsp; &nbsp;2.82842712&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; &nbsp;0.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ]</p>
<p>&nbsp;</p>
<p>Perform multidimensional scaling:</p>
<p>&nbsp;</p>
<p>mds = manifold.MDS(n_components=2, dissimilarity="precomputed", random_state=42,  metric = True)</p>
<p>results = mds.fit(disparities)</p>
<p>coords = results.embedding_</p>
<p>&nbsp;</p>
<p>cords =</p>
<p>&nbsp;[ -0.32503572&nbsp; &nbsp; &nbsp; 1.78399044 ]</p>
<p>&nbsp;[&nbsp; 0.09843686&nbsp; &nbsp; &nbsp; 0.87890749 ]</p>
<p>&nbsp;[&nbsp; 1.47842546&nbsp; &nbsp; &nbsp;-1.76761757 ]</p>
<p>&nbsp;[ -1.2518266&nbsp; &nbsp; &nbsp; -0.89528037 ]</p>
<p>&nbsp;</p>
<p>stress = results.stress_</p>
<p>0.0045113518633979315</p>
<p>&nbsp;</p>
<p>Now calculate stress by hand.</p>
<p>&nbsp;</p>
<p>First calculate Euclidian distances which correspond coordinates returned:</p>
<p>&nbsp;</p>
<p>dis = euclidean_distances(coords)</p>
<p>&nbsp;[ 0.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.9992518&nbsp; &nbsp; &nbsp; &nbsp;3.98326395&nbsp; &nbsp; &nbsp; 2.83503675&nbsp; ]</p>
<p>&nbsp;[ 0.9992518&nbsp; &nbsp;&nbsp;&nbsp;0.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;2.98470492&nbsp; &nbsp; &nbsp; 2.22956362&nbsp; ]</p>
<p>&nbsp;[ 3.98326395&nbsp; &nbsp;2.98470492&nbsp; &nbsp; &nbsp;0.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2.86622548&nbsp; ]</p>
<p>&nbsp;[ 2.83503675&nbsp; &nbsp;2.22956362&nbsp; &nbsp; &nbsp;2.86622548&nbsp; &nbsp; &nbsp; 0.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ]</p>
<p>&nbsp;</p>
<p>And now we can calculate stress value:</p>
<p>&nbsp;</p>
<p>stress =</p>
<p>((dis.ravel() - disparities.ravel()) ** 2).sum() / 2</p>
<p>0.0020293041020245147</p>
<p>&nbsp;</p>
<p>So, the real stress calculated using returned coordinates doesn&rsquo;t correspond to the stress value</p>
<p>results.stress_ = 0.0045113518633979315</p>
<p>&nbsp;</p>
<p>After some debugging it is clear that MDS returns coordinates for the current iteration and stress for the previous iteration.</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p><strong>Here is the script to reproduce:</strong></p>
<p>&nbsp;</p>
<p>from sklearn import manifold</p>
<p>from sklearn.metrics.pairwise import euclidean_distances</p>
<p>import pandas as pd</p>
<p>&nbsp;</p>
<p>data = [['A',1,5],['B',1,4],['C',1,1],['D',3,3]]</p>
<p>df = pd.DataFrame(data, columns = ['Name', 'X','Y'])</p>
<p>&nbsp;</p>
<p>#Distance matrix</p>
<p>disparities = euclidean_distances(df[['X','Y']])</p>
<p>mds = manifold.MDS(n_components=2, dissimilarity="precomputed", random_state=42,&nbsp; metric = True)</p>
<p>results = mds.fit(disparities)</p>
<p>coords = results.embedding_</p>
<p>stress = results.stress_</p>
<p>print ('returned stress=',stress)</p>
<p>&nbsp;</p>
<p># Calculate Stress by hand</p>
<p>dis = euclidean_distances(coords)</p>
<p>real_stress = ((dis.ravel() - disparities.ravel()) ** 2).sum() / 2</p>
<p>print('real stress =',real_stress)</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p><strong>Versions:</strong></p>
<p>System:</p>
<p>&nbsp;&nbsp;&nbsp; python: 3.7.6 (default, Dec 30 2019, 19:38:36)&nbsp; [Clang 10.0.0 (clang-1000.11.45.5)]</p>
<p>executable: /usr/local/opt/python/bin/python3.7</p>
<p>&nbsp;&nbsp; machine: Darwin-17.7.0-x86_64-i386-64bit</p>
<p>&nbsp;</p>
<p>Python deps:</p>
<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; pip: 19.3.1</p>
<p>setuptools: 42.0.2</p>
<p>&nbsp;&nbsp; sklearn: 0.21.3</p>
<p>&nbsp;&nbsp;&nbsp;&nbsp; numpy: 1.17.4</p>
<p>&nbsp;&nbsp;&nbsp;&nbsp; scipy: 1.3.2</p>
<p>&nbsp;&nbsp;&nbsp; Cython: 0.29.15</p>
<p>&nbsp;&nbsp;&nbsp; pandas: 1.0.1</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
user: akrupnik
issue_16836: state: open
title: Unexpected KNN kd_tree training times
body:  I've been analyzing the empirical performance scaling of the training times for a few different classifiers on the MNIST digits data set. I was surprised to find that KNN training times appear to grow faster than expected from theory (_i.e._, _n*log(n)_ -- the time that should be required to build the kd-tree).

To be precise, I collected some data on the training times of
`model = sklearn.neighbors.KNeighborsClassifier(weights='distance', algorithm='kd_tree', leaf_size=1, metric='minkowski', p=10)`
 for 2501 random subsets of the MNIST training dataset. The number of training instances in each training subset was varied from [2 500, 2 519, ..., 50 000]. 

I used the running time measurements from the smaller 30% of the training instance sets to fit a few running time scaling models to the data as a function of the number of training instances in each instance set. For example: `a*n*log(n) + b`, where `a` and `b` are the parameters of the running time model that I fit, and `n` is the number of training instances in each subset. The model that I fit was `2.50e-05*n*log(n) - 0.0247`. 

I then used this model to make predictions for the running times on the larger training instance sets (let's call these larger training instance sets "challenge data" for the models).  Surprisingly, the larger challenge running time data was quite a bit larger than model's predictions. I also performed some bootstrap analysis of the procedure to produce 95% confidence intervals for the running times of the challenge data and the model predictions, which quite clearly do not overlap with each other. 

I also performed this analysis with a polynomial model `a*n**b` and an exponential model `a*b**n` as references. The polynomial model fits `b=1.12`, and provides a better estimate for the challenge running times, but it also clearly under-estimates them. The exponential model, not surprisingly, clearly over-estimates the running times for the challenge data. You can see a plot of the resulting analysis for yourself below:

[fitted-models-knn-v0.22.2.pdf](https://github.com/scikit-learn/scikit-learn/files/4429763/fitted-models-knn-v0.22.2.pdf)

Note: There are confidence intervals for both the observed median running times as well as for each of the fitted models. Their shown as shaded regions using the same colours as the models. However, they don't show up well in the plot because they are very small -- there isn't a lot of variability in these running times that cannot be explained by the size of the training set.

I've tried tinkering with the experimental setup that I used to do this analysis to see if I could find a means of accounting for this performance. For example, I tried dropping some of the running times for the smaller training set sizes to see if perhaps there are some lower order terms that are affecting the models when fit to the smallest support sizes (e.g., perhaps some data structure initialization can initially dominate the running time costs). However, I've had no luck with any such attempts. They all produce similar results or leave me without enough data to learn anything that is statistically significant. I would think it could possibly also be attributed to the apparent jump discontinuities around each power of two; however, I would then expect it to be possible for the models to over-estimate the challenge data in some of these runs, but it never did that either.

Does anyone have any guesses as to what could cause the training time scaling to be greater than log-linear? From my understanding of the theoretical bounds of the algorithm, it shouldn't be able to scale worse than log-linear, so I wonder if this could be some kind of a performance bug -- or perhaps the theory isn't correctly accounting for a relevant cost that is reflected in the running times?
user: YashaPushak
issue_16834: state: open
title: [MRG] FEA Gower distance
body: Closes https://github.com/scikit-learn/scikit-learn/pull/9555
Fixes #5884

This PR continues the work done in #9555. Since the discussion on that PR is really long at this point, and this is a different implementation based on `scipy.spatial.distance.cdist`, I decided to create a new one to keep discussions separate.

Up until https://github.com/scikit-learn/scikit-learn/commit/4859b8115ddc7e99af00597efa637ddfbcd8d502 (at the time of opening this PR), there are minor changes to the existing tests. I'm planning to clean up the tests, but they were very helpful catching the issues with the edge cases, good job on that @marcelobeckmann.

However, I've changed a few design decisions:

- `categorical_features` is mandatory, and if not passed, the data is assumed to be numerical. If string columns are present, it'll simply fail to convert them to floats (a numpy error).
- `categorical_features` now also accepts a callable, which can be created using `make_column_selector` the same way as it's used in `ColumnTransformer`.
- It uses a `MinMaxScaler` instead of its own scaler.


Before I finalize the user guide and fix the tests, there are a few issues I'd like us to discuss, and I'll leave inline comments to start those discussions. 
user: adrinjalali
issue_16832: state: open
title: MultiOutputClassifier missleading score
body: https://github.com/scikit-learn/scikit-learn/blob/95d4f0841d57e8b5f6b2a570312e9d832e69debc/sklearn/multioutput.py#L421

Well, just to let you know that when using a MultiOutputClassifier I was finding the score results to be surprisingly low. After digging a bit, I found that the score of a MultiOutputClassifier was being calculated as the mean of y being equal to y_pred for all targets: 
`np.mean(np.all(y == y_pred, axis=1))` 

For me, that was not, at all, the expected behaviour when you state that score "_Returns the mean accuracy on the given test data and labels_"! Indeed, I was expecting the returned accuracy to consider all hits. Something like: 
`np.mean(np.mean(y == y_pred, axis=0))`

A simple example:
```
y = [['low', 'medium', 'medium'], ['medium', 'medium', 'medium'], ['medium', 'high', 'high']]
y_pred = [['medium', 'medium', 'medium'], ['medium', 'high', 'medium'], ['low', 'high', 'high']]
```
Would return an accuracy of 0 instead of 0.667!
Why not give the user the chance to choose?

Anyway, if you find the current behaviour to be the expected one, feel free to close this issue.
user: brunofmf
issue_16813: state: open
title: Support ERR (expected reciprocal rank) metric
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable

Calculating the ERR metric - http://olivier.chapelle.cc/pub/err.pdf (2009. currently 706 citations). Compared to nDCG at https://opensourceconnections.com/blog/2019/12/09/demystifying-ndcg-and-err/

#### Describe your proposed solution

Add this algorithm with the same API as [sklearn.metrics.ncdg_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ndcg_score.html#sklearn.metrics.ndcg_score)

#### Describe alternatives you've considered, if relevant

Implement it myself, include another library such as http://rankeval.isti.cnr.it/docs/rankeval.metrics.html.
- other libraries use different APIs
- other libraries may have questionable support or quality control

#### Additional context



user: brabster
issue_16805: state: open
title: Gaussian Process issue on training data
body: When I read the code of `scikit-learn/sklearn/gaussian_process/_gpr.py/` I faced a problem, I did not understand and I'm not sure whether this is some kind of error or a missunderstanding. Assuming I am creating a GP like this:

```
import numpy as np
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import (RBF)

k = RBF()
gp = GaussianProcessRegressor(kernel=k, optimizer=None, normalize_y=True)
X = np.array((1, 3))[:, np.newaxis]
y = np.array((10, 13))

gp.fit(X, y)
```

I would assume that `gp.y_train_` is afterwards `array([-1,  1])`, but it is in fact `array([-1.5,  1.5])`, as though in line 199 the division by standard deviation would not be performed:

`y = (y - self._y_train_mean) / self._y_train_std`

`gp.y_train_` is not being altered in the rest of the function, except in line 215, but regardless if `copy_X_train` is True or False, the values of y should be stored in `gp.y_train_`
user: mini1988
issue_16802: state: open
title: Feature Request: Include p-values attribute for logistic regression
body: `Scikitlearn` is the de facto home for all kinds of modeling algorithms. It has a plethora of
algorithms but still one thing that seems to be missing is the implementation of `LogisticRegression` where we can have p-values.


It would be great if we have something like `model.p_values_` attribute for the Logistic Regression Models.

I know that there is another statistical library `statsmodels` which provides p_values, but a lot of programmers use `sklearn` and they build models based on this library. It is somewhat
inconvenient to use `statsmodels` just to get p-values and run other models such as `Random Forest` in `sklearn`.

Afterall, the API of `statsmodels` and `sklearn` are quite different. `sklean` is trend setter and 
most people feel comfortable with `sklearn` API, however, `statsmodels` follows R-programming API and they are quite different.

In conclusion, It would be great if `sklearn` provides `p-values` for linear models.

I am eagerly waiting for the implementation in future versions of `sklearn`.
user: bhishanpdl
issue_16800: state: open
title: Add sparse support for 'y' to multioutput regression
body: #### Reference Issues/PRs
Fixes #16686 

#### What does this implement/fix? Explain your changes.
Now it's possible to use 'y' of type "sparse.csr_matrix" as input to MultiOutputRegressor.
user: Reksbril
labels: Needs Decision: Requires decision
issue_16792: state: open
title: Allow different scoring when "Some inputs do not have OOB scores." warning
body: #### Background
Random Forests: When the "Some inputs do not have OOB scores." warning is emitted during training, it appears that the scoring method calculates the score using the inputs with the OOB scores only. E.g. if calculating accuracy, it is calculated as the ratio of number of correctly classified inputs vs. the number of inputs with the OOB scores (non-nan values).
#### Proposed solution
It would be interesting to be able to get the accuracy calculated as the ratio of number of correctly classified inputs vs. the number of all inputs considered, probably by introducing a new parameter. This would automatically eliminate cases in GridSearchCV where the num_estimators is too low. This is currently more challenging to achieve. It would be also helpful to mention this behavior in the docs.
user: fdrobnic
issue_16789: state: open
title: Parameter n_classes in make_classification does not work as expected in extreme cases.
body: #### Describe the bug
According to the documentation, *n_classes* corresponds to the number of classes (or labels) of the classification problem. Therefore, I expected exactly n classes in the generated y. However, in at least one case, this did not work. I am well aware that the combination of parameters is absolutely stupid. This behavior was only discovered in a unit test of a wrapper function that was fed with random inputs.

#### Steps/Code to Reproduce
```
from sklearn.datasets import make_classification
import numpy as np

X, y = make_classification(n_samples=40, 
                         n_features=301,
                         n_informative=147,
                         n_redundant=22,
                         n_repeated=43,
                         n_classes=34, 
                         shuffle=False, 
                         random_state=527)

print(len(np.unique(y)))
```


#### Expected Results
The passed number of classes (*n_classes*):
`34`

#### Actual Results
`33`

#### Versions
System:
    python: 3.7.6 | packaged by conda-forge | (default, Mar  5 2020, 15:27:18)  [GCC 7.3.0]
executable: /opt/conda/bin/python
   machine: Linux-4.19.76-linuxkit-x86_64-with-debian-buster-sid

Python dependencies:
       pip: 20.0.2
setuptools: 46.0.0.post20200311
   sklearn: 0.22.2.post1
     numpy: 1.18.1
     scipy: 1.4.1
    Cython: 0.29.15
    pandas: 1.0.3
matplotlib: 3.1.3
    joblib: 0.14.1

Built with OpenMP: True

user: oXwvdrbbj8S4wo9k8lSN
issue_16765: state: open
title: return_all_estimators option in GridSearch and RandomizedSearch
body: #### Describe the workflow you want to enable

I would like to have an option in the GridSearchCV and RandomizedSearchCV
that enables storing of the (re)fitted estimator for all parameter combinations.

The background is that I'm working with highly correlated timeseries data and try to find the best  
way to set up cross validation.

I have a test set set aside and want to examine for which cross validation strategy, the CV score correlates best with the score on the test set.

Currently, only the "best" estimator is stored thus not allowing to evaluate the others on the test set.

#### Describe your proposed solution

A slight modification of BaseSearchCV should do the trick. I would give it a shot if appreciated and the above makes sense.

user: Thomasillo
issue_16756: state: open
title: `check_estimator` for estimators which require a mandatory extra fit param.
body: Right now the `check_estimators` will fail if the given estimator requires a parameter to `fit` other than `X, y`. https://github.com/fairlearn/fairlearn/pull/342 is an example of where it fails.

In some use-cases, it makes sense to require an extra parameter. For instance, in the case of fairness, you may require the `sensitive_features` to be passed as well as the data itself. How should we handle these cases?

An alternative would be to have feature names during fit and set the _sensitive feature names_ in `__init__`.

@thomasjpfan would you have an idea?
user: adrinjalali
issue_16750: state: open
title: Document/standardize how to skip network tests
body: Following discussion in https://github.com/scikit-learn/scikit-learn/issues/16743 I think it would be good to document and standardize how we skip tests that require network or download (this could also apply to tests that are skipped for other reason). Currently there are these mechanisms,
 1. `SKLEARN_SKIP_NETWORK_TESTS` for some tests
 2. `network` pytest marker for other tests, meaning that such tests can be deselected with `-m "not network"` pytest option
 3.  also the `--skip-network` pytest option, that skips tests with the `network` marker. The issue with this is that it requires a `conftests.py`, which is currently not available e.g. when scikit-learn is installed and tests are run with `pytest --pyargs sklearn`. 


I would propose to remove 1, and standardize/document the use of either 2 or 3. If we go with 3 we might need to move `conftests.py` under the `sklearn` folder, although it's also not ideal.
user: rth
labels: module:test-suite: everything related to our tests
issue_16721: state: open
title: Improve IsolationForest average depth evaluation
body: The equation in the original paper is undersimplified:

    c(n) = 2 H(n-1) - 2 (n-1) / n,

where H(k) ~ (ln(k) + gamma)

The definition of the harmonic number is:

    H(n) = sum_{j=1}^{n} (1 / j)

then it follows that:

    H(n) = H(n-1) + 1 / n

and then:

    c(n) = 2 H(n-1) - 2 (n-1) / n = 2 ( H(n-1) - 1 + 1 / n ) =
         = 2 (H(n) - 1)

Here we use simplier equation to save calculations.

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: matwey
issue_16715: state: open
title: Doc/Discussion: Discrimination in examples and user guide
body: #### Goal
A space to discuss thougtfully and amicably, if the topic of discrimination and bias should be addressed in the documentation (examples and user guide):
- bias in model
- bias in data

#### Non Goal
Endless and pointless discussions. So please, use every word wisely and sparingly.

#### Possible Solutions
Mention in example X that the data may contain bias and reference a link to a good source for further insights.
Discover model bias in example Y for a certain feature subspace and investigate a little.

#### References
The examples [Poisson regression and non-normal loss](https://scikit-learn.org/dev/auto_examples/linear_model/plot_poisson_regression_non_normal_loss.html#sphx-glr-auto-examples-linear-model-plot-poisson-regression-non-normal-loss-py) and [Tweedie regression on insurance claims](https://scikit-learn.org/dev/auto_examples/linear_model/plot_tweedie_regression_insurance_claims.html#sphx-glr-auto-examples-linear-model-plot-tweedie-regression-insurance-claims-py) use an insurance dataset, example [Common pitfalls in interpretation of coefficients of linear models](https://scikit-learn.org/dev/auto_examples/inspection/plot_linear_model_coefficient_interpretation.html) uses a wages dataset.
Some concern where pointed out in https://github.com/scikit-learn/scikit-learn/pull/16648#discussion_r388974502.
user: lorentzenchr
issue_16714: state: open
title: [MRG] Allow only fit_transform to be present in pipeline
body: This makes it possible to plug in algorithms like TSNE in the
pipeline, although they do not support `transform`.  A warning will be
raised that the pipeline is not reusable.

#### Reference Issues/PRs
Fixes  #16710

#### What does this implement/fix? Explain your changes.
This allows algorithms that only implement a `fit_transform` function, but not `transform` (such as TSNE) to be used in a Pipeline in sklearn.

user: jnboehm
labels: Needs Decision: Requires decision
issue_16710: state: open
title: Pipeline requires both fit and transform method to be available instead of only fit_transform
body: 
#### Describe the bug
Calling a pipeline with a nonparametric function causes an error since the function `transform()` is missing.  The pipeline itself calls the function `fit_transform()` if it's present.  For nonparametric functions (the most prominent being t-SNE) a regular `transform()` method does not exist since there is no projection or mapping that is learned.  It could still be used for dimensionality reduction.

#### Steps/Code to Reproduce
Example:
```python
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.pipeline import make_pipeline

make_pipeline(TSNE(), PCA())
```


#### Expected Results
A pipeline is created.

#### Actual Results
Output:
```
TypeError: All intermediate steps should be transformers and implement fit and transform or be the string 'passthrough' 'TSNE(angle=0.5,...
```
#### Possible Solution
Editing this https://github.com/scikit-learn/scikit-learn/blob/95d4f0841/sklearn/pipeline.py#L179
to the following in order to reflect the change:
```python
if (not hasattr(t, "fit_transform")) or not (hasattr(t, "fit") and hasattr(t, "transform")):
```

#### Versions
```
import sklearn; sklearn.show_versions()
System:
    python: 3.8.2 (default, Feb 26 2020, 22:21:03)  [GCC 9.2.1 20200130]
executable: /usr/bin/python3
   machine: Linux-5.5.9-arch1-2-x86_64-with-glibc2.2.5

Python dependencies:
       pip: 20.0.2
setuptools: 46.0.0
   sklearn: 0.22.2.post1
     numpy: 1.18.1
     scipy: 1.4.1
    Cython: 0.29.15
    pandas: 1.0.1
matplotlib: 3.2.0
    joblib: 0.14.1

Built with OpenMP: True
```

<!-- Thanks for contributing! -->

user: jnboehm
labels: Low Priority: Low priority issues and pull requests
issue_16705: state: open
title: Support typing
body: #### Describe the workflow you want to enable

Support Python typing, which is available in Python 3.5+, compatible with scikit-learn. This can enable tools to automatically get the estimator arg types (e.g., linters), for example. And it's a Python standard now.

#### Describe your proposed solution

For example, `SVC` could be like this:

```python
def __init__(
    self,
     C: float = 1.0,
     kernel: Union[Callable[[np.ndarray, np.ndarray], np.ndarray],
                   Literal["linear", "poly", "rbf", "sigmoid", "precomputed"]] = "rbf",
     degree: int = 3,
     gamma: Union[float, Literal["scale", "auto"]] = "scale",
     coef0: float = 0.0,
     shrinking: bool = True,
     probability: bool = False,
     tol: float = 1e-3,
     cache_size: int = 200,
     class_weight: Optional[Union[Mapping[int, float], Literal["balanced"]]] = None,
     verbose: bool = False,
     max_iter: int = -1,
     decision_function_shape: Literal["ovo", "ovr"] = "ovr",
     break_ties: bool = False,
     random_state: Optional[Union[int, RandomState]] = None,
 ) -> None:
```

#### Describe alternatives you've considered, if relevant

An alternative is to keep the types just in the docstrings. However, is hard for tools to parse the arg types.
user: bryant1410
issue_16694: state: open
title: Loss Function (All-Threshold Loss) for ordinal response variables HistGradientBoosting
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
If the target is discrete multiclass, but ordinal (ordered) in nature (e.g. Likert scale, user ratings, preference levels), as opposed to nominal, would be great to have support for loss function that recognize the ordinal nature of the target variable.

(Note the difference between "learning-to-rank" and "ordinal classification". "learning-to-rank", implemented by popular boosting libraries XGBoost, LightGBM, CatBoost, produces relative ordering between items, and the output predictions are not class labels, while "ordinal classification" does produce class labels. Here is a blog post to introduce the reader to [loss functions for ordinal classification](http://fa.bianp.net/blog/2013/loss-functions-for-ordinal-regression/).)
#### Describe your proposed solution
All-Threshold loss is the proposed solution. [Rennie, et al](https://ttic.uchicago.edu/~nati/Publications/RennieSrebroIJCAI05.pdf) described the All-Threshold loss function. Here is an easy-to-understand blog post introducing readers to [loss functions for ordinal classification](http://fa.bianp.net/blog/2013/loss-functions-for-ordinal-regression/).
Fabian, et al implemented the [mord](https://github.com/fabianp/mord) package using logistic regression with All Threshold Loss

#### Describe alternatives you've considered, if relevant
Historically, in Logistic Regression, popular approach proposed by McCullagh ([paper](https://www.jstor.org/stable/2984952)) uses Cumulative Logit link function. Here is the main disadvantage of this approach:
> Probabilistic models for discrete ordinal response have also
been studied in the statistics literature [McCullagh, 1980;
Fu and Simpson, 2002]. However, the models suggested are
much more complex, and even just evaluating the likelihood
of a predictor is not straight-forward.
From Page 2 Rennie, Jason D. M. and Srebro, Nathan . [Loss functions for preference levels: Regression with discrete ordered labels](https://ttic.uchicago.edu/~nati/Publications/RennieSrebroIJCAI05.pdf).

Note that Cumulative Logit is the probabilistic approach, proposed by McCullagh in 1980.
All threshold loss is much easier to scale to large datasets, which is very much the purpose of HistGradientBoosting.
#### References
Papers
[1] Rennie, Jason D. M. and Srebro, Nathan, Loss functions for preference levels: Regression with discrete ordered labels.
[2] Dembczynski, Krzysztof and Kotlowski, Wojciech and Slowinski, Roman. Ordinal Classification with Decision Rules.
[3] Pedregosa, Fabian and Bach, Francis R. and Gramfort, Alexandre. On the Consistency of Ordinal Regression Methods.
[4] Li, Ling and Lin, Hsuan-Tien. Ordinal Regression by Extended Binary Classification.

user: hrbzkm98
issue_16686: state: open
title: MultiOutputRegressor doesn't support sparse y matrix
body: This snippet works fine because the sparse `y` is converted to a numpy array. 
```
from sklearn import linear_model, multioutput
from scipy.sparse import csr_matrix

x = csr_matrix((13, 5))
y = csr_matrix((13, 2))

m_reg = multioutput.MultiOutputRegressor(linear_model.SGDRegressor())
m_reg.fit(x, y.toarray())
```
Same snippet without converting the sparse `y` matrix:
```
from sklearn import linear_model, multioutput
from scipy.sparse import csr_matrix

x = csr_matrix((13, 5))
y = csr_matrix((13, 2))

m_reg = multioutput.MultiOutputRegressor(linear_model.SGDRegressor())
m_reg.fit(x, y)
```

This is the traceback:
```
/apps/python3/lib/python3.7/site-packages/sklearn/utils/validation.py in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, warn_on_dtype, estimator)
    758                         dtype=None)
    759     else:
--> 760         y = column_or_1d(y, warn=True)
    761         _assert_all_finite(y)
    762     if y_numeric and y.dtype.kind == 'O':

/apps/python3/lib/python3.7/site-packages/sklearn/utils/validation.py in column_or_1d(y, warn)
    795         return np.ravel(y)
    796 
--> 797     raise ValueError("bad input shape {0}".format(shape))
    798 
    799 

ValueError: bad input shape ()
```

My reading of the source is that the line [here](https://github.com/scikit-learn/scikit-learn/blob/95d4f0841d57e8b5f6b2a570312e9d832e69debc/sklearn/utils/validation.py#L785 ) does not convert a sparse matrix into a numpy array. That is
```
np.asarray(csr_matrix((13, 2)))
```
does not return a numpy array of shape `(13, 2)`, instead it is a shape `()` array containing the sparse matrix as its only element.
user: allentran
issue_16677: state: open
title: List tests that are skipped or xfailed in all CI jobs
body: It would be useful to list tests that are skipped or xpass (i.e. marked as a failure but actually pass) in all CI jobs.

I think someone mentionned @jeremiedbb you had done it in another project?
user: rth
issue_16668: state: open
title: Tweedie deviance loss for tree based models
body: #### Describe the workflow you want to enable
If the target `y` is (approximately) Poisson, Gamma or else Tweedie distributed, it would be beneficial for tree based regressors to support [Tweedie deviance loss functions](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-poisson-gamma-and-tweedie-deviances) as splitting criterion. This partially addresses #5975.

#### Describe your proposed solution
Ideally, one first implements
- [ ] differentiable loss functions #15123

and then adds the different loss criteria to the tree based models:
- [ ]  `DecisionTreeRegressor`
- [ ] `RandomForestRegressor`
- [ ] `GradientBoostingRegressor`
- [ ] `HistGradientBoostingRegressor`.

#### Open for Discussion
For Poisson and Tweedie deviance with `1<=power<2`, ther target `y` may be zero while the prediction `y_pred` must be strictly larger than zero. A tree might find a split where one node has  `y=0` for all samples in that node, resulting naively in `y_pred = mean(y) = 0` for that node. I see 3 different solutions to that:
1. Use a log-link function, i.e. predict `y_pred = np.exp(tree)`
    See #16692 for HistGradientBoostingRegressor. This may be no option for DecisionTreeRegressor.
2. Use a splitting rule that forbids splits where one node has `sum(y)=0`.
    One might also introduce some option like `min_y_weight`, such that splits with  `sum(sample_weight*y) < min_y_weight` are forbidden.
3. Use some form of parent child average `y_pred = a * mean(y) + (1-a) * y_pred_parent` and forbid further splits, see [1].
(Bayes/credibility theory motivates to set `a = sum(sample_weight*y)/(gamma+sum(sample_weight*y))` for some hyperparameter `gamma`.)

There is also a dirty solution that allows `y_pred=0` but sets the value `min(eps, y_pred)` in the loss function for some tiny value of `eps`.

#### References
[1] [R rpart library](https://cran.r-project.org/web/packages/rpart/vignettes/longintro.pdf), chapter 8 Poisson regression
user: lorentzenchr
issue_16666: state: open
title: Top-k accuracy - Add support for multilabel/multioutput cases
body: #### Describe the workflow you want to enable

Add support for multilabel/multioutput top-k accuracy. Multiclass setting being implemented here #16625 

#### Additional context

- Some multilabel classifiers treat each label independently, when it's the case does it make sense to rank independent predicted scores?

- Multioutput-multiclass approach treats each output as a multiclass problem, therefore we could consider an answer for a sample as correct when all outputs are correct.

user: gbolmier
labels: Needs Decision: Requires decision
issue_16658: state: open
title: Adding a fprintf to svm kernel function dot will cause precision problem on ubuntu 32
body: #### Describe the bug
This bug is related to PR #16530 
I find a simple bug in svm.cpp, that adding a fprintf function to svm dot will cause precision problem on [32-bit ubuntu](https://hub.docker.com/layers/i386/ubuntu/18.04/images/sha256-4d07b5b0cd47c06a3ca847536a3e05901c6bf9d9f52dbb0e6a7fff9141453f11?context=explore) with [dependency](https://github.com/scikit-learn/scikit-learn/blob/4c29be44facbdaef188f84bdc8bf1190b2eebe07/build_tools/azure/install.sh#L89). Plus, it can pass the precision test on 64-bit ubuntu. Here is the code (only 3 lines).

#### Steps/Code to Reproduce
```
From 3c903b79aba97f13225255f4b395066d3c90bc9f Mon Sep 17 00:00:00 2001
From: Shuahua Fan <shuhua.fan@intel.com>
Date: Fri, 6 Mar 2020 11:26:09 +0800
Subject: [PATCH] simple addition for logging to file

---
 sklearn/svm/src/libsvm/svm.cpp | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/sklearn/svm/src/libsvm/svm.cpp b/sklearn/svm/src/libsvm/svm.cpp
index 9321340..09fa935 100644
--- a/sklearn/svm/src/libsvm/svm.cpp
+++ b/sklearn/svm/src/libsvm/svm.cpp
@@ -406,6 +406,9 @@ double Kernel::dot(const PREFIX(node) &px, const PREFIX(node) &py)
 	int dim = min(px.dim, py.dim);
 	for (int i = 0; i < dim; i++)
 		sum += px.values[i] * py.values[i];
+	FILE* m_file = fopen("/dev/null", "a");
+	fprintf(m_file, "test\n");
+	fclose(m_file);
 	return sum;
 }
 #else
-- 
2.17.1
```

And the bug report is as follows.
```
cd scikit-learn-0.22.1/sklearn/ensemble/tests
python3
>>> import test_bagging as tt
>>> tt.test_sparse_classification()
/usr/local/lib/python3.6/dist-packages/sklearn/ensemble/_bagging.py:783: RuntimeWarning: divide by zero encountered in log
  return np.log(self.predict_proba(X))
/usr/local/lib/python3.6/dist-packages/sklearn/ensemble/_bagging.py:783: RuntimeWarning: divide by zero encountered in log
  return np.log(self.predict_proba(X))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/ubuntu1804/scikit-learn-0.22.1-simple/sklearn/ensemble/tests/test_bagging.py", line 132, in test_sparse_classification
    assert_array_almost_equal(sparse_results, dense_results)
  File "/usr/lib/python3/dist-packages/numpy/testing/utils.py", line 962, in assert_array_almost_equal
    precision=decimal)
  File "/usr/lib/python3/dist-packages/numpy/testing/utils.py", line 778, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 6 decimals

(mismatch 66.66666666666666%)
 x: array([[-0.219957,  0.984658,  2.232507],
       [-0.218276,  1.671745,  1.573845],
       [-0.223031,  1.780723,  1.525337],...
 y: array([[-0.219949,  0.984647,  2.232505],
       [-0.218277,  1.671742,  1.573849],
       [-0.223032,  1.780719,  1.52534 ],...

```

#### Versions
```
>>> import sklearn; sklearn.show_versions()

System:
    python: 3.6.9 (default, Nov  7 2019, 10:44:02)  [GCC 8.3.0]
executable: /usr/bin/python3
   machine: Linux-5.4.4-879.native-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
       pip: 9.0.1
setuptools: 39.0.1
   sklearn: 0.22.1
     numpy: 1.13.3
     scipy: 0.19.1
    Cython: 0.29.15
    pandas: None
matplotlib: 2.1.1
    joblib: 0.14.1

Built with OpenMP: True
```


<!-- Thanks for contributing! -->

user: jim0421
issue_16653: state: open
title: check_is_fitted and validate_date are a performance bottlenck for ensembles prediction
body: Hi all,

I am doing some profiling for random forest classifiers prediction.

A simple predict_proba for a single input on an RF calls a check_is_fitted on the RF and then calls predict_proba on all 100 internal trees which in turn call check_is_fitted on each tree, etc ... check_is_fitted is simply not adapted for ensembles. check_is_fitted is called hundreds of times in sklearn code as if it was a non-costly function.

My understanding is that if an RF is fitted (fit() called once), then we can without a lot of risk assume that all its internal estimators (trees) are already fitted ;)

In my case, the performance problem is that the current implementation of check_is_fitted is independent of the estimator used (validation.py) and calls some generic functions on the python object and checks if an attribute with some pattern of variable naming is present (some kind of tribal knowledge if not a simple convention). Listing all members can be costly : depends on the type and the complexity of the estimator (=> performance issue).

Some possible fix is to make each estimator aware of the fact that it is already fitted (RF checks the presence of the member 'estimators_' and nothing more, the tree checks the member 'tree_', the Ridge checks 'coef_' etc). A new service estimator.is_fitted() can be added and implemented for each estimator (not that much work for a boolean method).

Please get rid of this function !!!

Thanks in advance,
Antoine

user: antoinecarme
issue_16650: state: open
title: Improved calibration example plots
body: Suggested improved calibration plots with density plots, which I thought are easier to tell where the peaks are than histograms.

https://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_curve.html
and
https://scikit-learn.org/stable/auto_examples/calibration/plot_compare_calibration.html

The new ones would look like
![plot_calibration_curve_1](https://user-images.githubusercontent.com/527862/76040329-e6cda980-5f03-11ea-8cfa-d15d3e6cd0e4.png)
![plot_calibration_curve_2](https://user-images.githubusercontent.com/527862/76040333-e9300380-5f03-11ea-8fd0-c337390605fe.png)
![plot_compare_calibration](https://user-images.githubusercontent.com/527862/76040335-e9c89a00-5f03-11ea-8569-e302d25b3d66.png)




user: zyxue
issue_16649: state: open
title: Custom Learner for AdaboostClassifier
body: Hello, i'd like to use AdaboostClassifier with a base_estimator that doensn't implement `sample_weight` function. I'd like to know the proper way of implementing this functions. If they vary from one classifier to another, then i may say that i want to use MLPClassifier as base_estimator.

Thanks,
Finder Lars
user: vendrafilm
issue_16643: state: open
title: _unnormalized_transform request
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
Now, sklearn LDA only supports normalized transform output, which is the `transform`  method. But I still need to use the non-normalized output, so could you please open this feature?

#### Describe your proposed solution
open this `_unnormalized_transform` as an official method.

Thanks

user: Wuliyuanulb
issue_16642: state: open
title: SVC with ADABoosting
body: 

I am trying to use SVC with ADAboosting. WIth SVC, but not other base estimators, the initial estimator does not seemed to be trained.

Comparing the initial estimator of the ensemble to a single estimator with the same hyper-parameters 

create an ADA ensemble

<img width="986" alt="Screen Shot 2020-03-04 at 9 15 10 PM" src="https://user-images.githubusercontent.com/19198429/75944794-39ed2100-5e5e-11ea-9125-b06cfaf729d5.png">

Create a single SVC

<img width="553" alt="Screen Shot 2020-03-04 at 9 15 20 PM" src="https://user-images.githubusercontent.com/19198429/75944859-58ebb300-5e5e-11ea-9f3c-65c098b4a9f4.png">

Compare the accuracies

<img width="415" alt="Screen Shot 2020-03-04 at 9 15 49 PM" src="https://user-images.githubusercontent.com/19198429/75944878-630db180-5e5e-11ea-9760-a38601e0d873.png">


I have tried different hyper-parameters and had the same issue. This issue does happen for RandomForests and DecisionTree Classifiers



user: QuantumChamploo
issue_16637: state: open
title: Add L1 penalty and coordinate descent solver to TweedieRegression
body: The L1 penalty with  coordinate descent solver for GLM was proposed in https://github.com/scikit-learn/scikit-learn/pull/9405 and was not included in the minimal implementation merged in https://github.com/scikit-learn/scikit-learn/pull/14300.

It would be be good to add it, however care must to taken to avoid redundancy with the existing coordinate descent solver in ElasticNet.

Preliminary benchmarks were done in https://github.com/scikit-learn/scikit-learn/pull/9405#issuecomment-499578511

Also see: https://github.com/scikit-learn/scikit-learn/issues/16634 https://github.com/scikit-learn/scikit-learn/issues/16635

user: rth
issue_16608: state: open
title: Visualization and validation tools for regression
body: Currently most of the plotting tools available in scikit-learn are related to classification (https://scikit-learn.org/stable/modules/classes.html#id3). It would be good to add more visualizers for regressions.

Plotting y_true as a function of y_pred, or the residuals, as done frequently is simple enough and probably doesn't need a dedicated plot function.

Other approaches worth considering were discussed in https://github.com/scikit-learn/scikit-learn/pull/10084,
 - [Lorentz curves](https://en.wikipedia.org/wiki/Lorenz_curve#Definition_and_calculation), and the associated Gini coefficient computed as AUC:  see [example](https://95549-843222-gh.circle-artifacts.com/0/doc/auto_examples/linear_model/plot_tweedie_regression_insurance_claims.html) in https://github.com/scikit-learn/scikit-learn/pull/14300. This was earlier proposed https://github.com/scikit-learn/scikit-learn/pull/10084 and more recently in https://github.com/scikit-learn/scikit-learn/pull/15176 (and in particular https://github.com/scikit-learn/scikit-learn/pull/15176/commits/a608c703b5960994c569e0191775c3dfa0c805e0). It's more of a ranking metric than a regression metric.

 - The name varies, but the idea is to select a numerical feature, bin it, and compare the mean of y_true with the mean of y_pred for each bin. This allows to evaluate how well the model performs for different sub-categories of data. Example in https://95549-843222-gh.circle-artifacts.com/0/doc/auto_examples/linear_model/plot_poisson_regression_non_normal_loss.html Somewhat related to partial dependence plots.

There are a few additional ones in [yellowbrick](https://www.scikit-yb.org/en/latest/api/regressor/index.html), I'm not convinced they would be that useful in scikit-learn.

Not saying we need all of them, but just wanted to move this discussion from outside the GLM PR.
user: rth
issue_16601: state: open
title: Tags for the example gallery?
body: Just throwing out the idea...

We have lots of examples in the gallery. Some of them are beginner oriented like https://github.com/scikit-learn/scikit-learn/pull/15706, and some are much more advanced, like the ones proposed in https://github.com/scikit-learn/scikit-learn/pull/14300. It can be overwhelming for newcomers who don't necessarily know what they're looking at.

**How would you feel about adding tags**, e.g. "Beginner", "Advanced", "New in 0.XX" etc.? I'm picturing these as labels e.g. on the rendered image in the gallery, but we could also use them for filtering?

I guess this is not just scikit-learn related but also a sphinx-gallery feature,  I'm just testing the water here first
user: NicolasHug
issue_16600: state: open
title: Output of OneHotEncoder fed into CCA throws ValueError: The output of the '<pipeline_name>' transformer should be 2D (scipy matrix, array, or pandas DataFrame).
body: #### Describe the bug
Output of OneHotEncoder fed into CCA throws ValueError: The output of the '<pipeline_name>' transformer should be 2D (scipy matrix, array, or pandas DataFrame)

Notes:
Please note that this is not an issue if the above is done not in a pipeline ie. I call fit_transform on the OneHotEncoder and then use the output with CCA.fit_transform. The issue occurs if the entire Column Transformer pipeline is used.

#### Steps/Code to Reproduce
1. Create a pipeline transformer, where input is one hot encoded with OneHotEncoder, its output is fed to CCA.
2. Run fit_transform on the data using the pipeline. 
3. Should return a numpy array with number of features specified in CCA

```
Sample code to reproduce the problem
```

```
categorical = [<some categorical column names from pandas dataframe>]
numerical = [<some numerical column names from pandas dataframe>]

ohe_pipe =  ColumnTransformer(
    [
    ("one_hot_encode", Pipeline([("one_hot_encode", OneHotEncoder(handle_unknown="ignore", sparse=False)),  ("cca", CCA(n_components=300))]), categorical),
("scale", StandardScaler(), numerical),
    ],
    remainder="drop")

X_test = ohe_pipe.fit_transform(X_test, y_test)

```

#### Expected Results
Should produce a numpy array such that if `X_test.shape` is called the output should be 
```
>>>(<num examples>, 300)
```

#### Actual Results
```
D:\Users\<USERNAME>\Anaconda3\lib\site-packages\sklearn\compose\_column_transformer.py in fit_transform(self, X, y)
    536 
    537         self._update_fitted_transformers(transformers)
--> 538         self._validate_output(Xs)
    539 
    540         return self._hstack(list(Xs))

D:\Users\<USERNAME>\Anaconda3\lib\site-packages\sklearn\compose\_column_transformer.py in _validate_output(self, result)
    400                 raise ValueError(
    401                     "The output of the '{0}' transformer should be 2D (scipy "
--> 402                     "matrix, array, or pandas DataFrame).".format(name))
    403 
    404     def _validate_features(self, n_features, feature_names):

ValueError: The output of the 'one_hot_encode' transformer should be 2D (scipy matrix, array, or pandas DataFrame).
```

#### Versions
```
System:
    python: 3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)]
executable: D:\Users\<USERNAME>\Anaconda3\python.exe
   machine: Windows-10-10.0.17763-SP0

Python dependencies:
       pip: 20.0.2
setuptools: 45.2.0.post20200210
   sklearn: 0.22
     numpy: 1.18.1
     scipy: 1.4.1
    Cython: None
    pandas: 0.25.1
matplotlib: 3.1.3
    joblib: 0.14.1

Built with OpenMP: True
```
(The reason that these versions are used and not newest is due to a memmapping problem experienced in windows) 

user: izabella197
labels: Needs Decision - API: Requires decision regarding API
issue_16593: state: open
title: OneHotEncoder.get_feature_names doesn't work with integer column names
body: ```python
import pandas as pd
import numpy as np
from sklearn.preprocessing import OneHotEncoder
X = pd.DataFrame({1: np.random.randint(0, 10, size=400)})
OneHotEncoder().fit(X).get_feature_names([1])
```
```pythontb
TypeError: unsupported operand type(s) for +: 'int' and 'str'
```
user: amueller
labels: Needs Decision: Requires decision
issue_16592: state: open
title: Reuse cache-aligned euclidean_distances in distance trees
body: I'm thinking it might be worthwhile to try to replace the euclidean distances used in kd-tree and ball-tree here:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/_dist_metrics.pxd#L18

With the cache-aligned one that @jeremiedbb wrote for kmeans:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/cluster/_k_means_fast.pyx#L31

Wdyt?
user: amueller
issue_16581: state: open
title: Implement Extended Isolation Forest
body: #### Reference Issues/PRs
Closes #16517 
(note that the feature proposal wasn't approved yet, but well, I need it for my job :) )

#### What does this implement/fix? Explain your changes.
It implements the Extended Isolation Forest Algorithm by:
- Adding an "extended" argument to the Isolation Forest class,
- Add a new "random_oblique" mode to the ExtraTreeRegressor class, which uses a new splitter,
- Add this new RandomObliqueSplitter to split instances with an oblique hyperplane.

#### Any other comments?

This is a WIP, several things still need to be implemented or require a discussion:
- The sparse version of the splitter is not implemented yet,
- What should be done with the `max_features` , `min_samples_leaf` and `min_weight_leaf` arguments needs to be discussed. With the oblique splitter, there is not really a way to select the best split out of several ones, unless we use `max_features` as the number of retries to find the best split ? The other two parameters could then be used to drop incorrect split then.

user: groud
issue_16574: state: open
title: WIP Enabling different array types (CuPy) in PCA with NEP 37
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
After discussing with @seberg, it was concluded that a prototype of using CuPy with sklearn can help with seeing how libraries would use NEP 37.

- [NEP 37](https://numpy.org/neps/nep-0037-array-module.html)
- https://github.com/scikit-learn/scikit-learn/issues/11447
- https://github.com/scikit-learn/scikit-learn/pull/14963
- Using `enable_duck_array` as a config flag for now

#### What does this implement/fix? Explain your changes.
For reference, this is how this will look like for the user:

https://gist.github.com/thomasjpfan/da54f874f3d434eb4ae360b5e6fa4c12

### With numpy

```py
import numpy as np
from sklearn.decomposition import PCA

rng = np.random.RandomState(42)
X = rng.rand(15, 5)

pca = PCA(n_components=3, random_state=0)
X_trans = pca.fit_transform(X)
type(X_trans)
# numpy.ndarray
```

### With cupy

```py
X_cp = cupy.asarray(X)
type(X_cp)
# cupy.core.core.ndarray

X_cp_trans = pca.fit_transform(X_cp)
# ValueError

with sklearn.config_context(enable_duck_array=True):
    X_cp_trans = pca.fit_transform(X_cp)
type(X_cp_trans)
# cupy.core.core.ndarray
```


#### Any other comments?
This is a quick prototype on how we can use NEP 37 in PCA. (There is still work for other solvers)
Note the `get_array_module` is a hack for now. It would be called with numpy if NEP 37 gets accepted.

CC: @amueller @seberg

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
issue_16561: state: open
title: categoricalNB fails with sparse matrix
body: According to the docs CategoricalNB requires X to be "array-like, sparse matrix}, shape = [n_samples, n_features]". However if I try to fit with a sparse matrix I get an error saying it has to be dense. It works fine passing a sparse matrix to fit a MultinomialNB model:

TypeError                                 Traceback (most recent call last)
<ipython-input-19-afb2039f4bc4> in <module>
      1 from sklearn.naive_bayes import CategoricalNB
      2 m = CategoricalNB()
----> 3 m.fit(x, y)
      4 preds = m.predict(x)
      5 m.score(x, y)

~\Anaconda3\lib\site-packages\sklearn\naive_bayes.py in fit(self, X, y, sample_weight)
   1101         self : object
   1102         """
-> 1103         return super().fit(X, y, sample_weight=sample_weight)
   1104 
   1105     def partial_fit(self, X, y, classes=None, sample_weight=None):

~\Anaconda3\lib\site-packages\sklearn\naive_bayes.py in fit(self, X, y, sample_weight)
    607         self : object
    608         """
--> 609         X, y = self._check_X_y(X, y)
    610         _, n_features = X.shape
    611         self.n_features_ = n_features

~\Anaconda3\lib\site-packages\sklearn\naive_bayes.py in _check_X_y(self, X, y)
   1161         # X, y = check_array(X, y, dtype='int', accept_sparse=False,
   1162         #                    force_all_finite=True)
-> 1163         X, y = check_X_y(X, y, accept_sparse=False, force_all_finite=True)
   1164         X, y = check_X_y(X, y, dtype='int')
   1165         if np.any(X < 0):

~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, warn_on_dtype, estimator)
    753                     ensure_min_features=ensure_min_features,
    754                     warn_on_dtype=warn_on_dtype,
--> 755                     estimator=estimator)
    756     if multi_output:
    757         y = check_array(y, 'csr', force_all_finite=True, ensure_2d=False,

~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)
    509                                       dtype=dtype, copy=copy,
    510                                       force_all_finite=force_all_finite,
--> 511                                       accept_large_sparse=accept_large_sparse)
    512     else:
    513         # If np.array(..) gives ComplexWarning, then we convert the warning

~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in _ensure_sparse_format(spmatrix, accept_sparse, dtype, copy, force_all_finite, accept_large_sparse)
    304 
    305     if accept_sparse is False:
--> 306         raise TypeError('A sparse matrix was passed, but dense '
    307                         'data is required. Use X.toarray() to '
    308                         'convert to a dense numpy array.')

TypeError: A sparse matrix was passed, but dense data is required. Use X.toarray() to convert to a dense numpy array.
user: simonm3
issue_16548: state: open
title: Hierarchical Precision, Recall, and F-Score
body: (if there is interest in this feature, I'm also interested in implementing it via PR based on guidance here)

#### Describe the workflow you want to enable
When dealing with hierarchical classification tasks, it is sometime useful to have metrics that take into account the class hierarchy.  For example, if classifying clothing like:

                     Clothing
                 /                   \
        Shirts                     Pants
        /    \                        /    \
T-Shirt   Sweatshirt   Jeans   Yoga Pants

Assigning a T-Shirt to Yoga Pants may be worse than to Sweatshirt because T-Shirt and Sweatshirt are still both shirts.  To account for this, an implementation of hierarchical metrics is proposed.

#### Describe your proposed solution
A hierarchical implementation of precision, recall, and f-score are proposed.  These metrics are analogous to their non-hierarchical versions except they consider the steps made within the hierarchy rather than just the final classification.   For example, recall is the number of correct steps taken while making predictions divided by the total number of steps required to make all predictions correctly.  Descriptions of these metrics can be found [here](https://idalab.de/blog/data-science/hierarchical-metrics) and [here](https://towardsdatascience.com/keeping-it-classy-how-quizlet-uses-hierarchical-classification-to-label-content-with-academic-4e89a175ebe3).  

Proposed here is either a separate implementation of these metrics alongside current implementations (eg: `hierarchical_precision_recall_fscore_support`), or implementation within the existing metrics.  Integration in existing metrics could be achieved through an additional `hierarchy` parameter for relevant functions.  If `hierarchy==None` then existing behavior is observed, but if `hierarchy` is defined as something that describes a hierarchy (for example, input suitable for a networkx graph?) then hierarchical scoring will be applied.  Averaging techniques (micro, macro, etc) will be unaffected as they apply to both implementations.  

#### Describe alternatives you've considered, if relevant
Implementation elsewhere in scikit-learn is possible.  Leaving it out of scikit-learn is of course an option.  But, while not exceedingly complex, the implementation of these metrics is non-trivial and easy to make mistakes on. 
user: ca-scribner
issue_16540: state: open
title: Easy access to citations for specific methods
body: Hello,

As a scientist, I would appreciate an easy access to citation information for a specific estimator. A very good example would be the citation function from R
https://www.rdocumentation.org/packages/utils/versions/3.6.2/topics/citation
That provides citation for each packages (and even better in bibtex format !).

The citation information in sklearn is more or less available in the reference sections of the doctrings and i coded a short function that basically prints the section:

```python
import re
import textwrap

def _get_docstring_sections(txt):
    """Returns the sections content of numpy formated docstrings as dict."""
    
    if not type(txt)==str:
        txt=''
    
    # remove docstring indentation
    txt=textwrap.dedent('    '+txt)

    # split the docstring by sections (------)
    split_sec=re.split('\n--*\n',txt)
    
    # Find fisrt title
    secs={}
    lines=split_sec[0].splitlines()
    if len(lines):
        title= lines[-1]
        for i in range(1,len(split_sec)):
            lines=split_sec[i].splitlines()
            secs[title]='\n'.join(lines[:-1])
            title=lines[-1]
        
    return secs

def cite(obj=None,format=None):
    """Get citation information from object with numpy docstrings
    
    """

    txt="""No references found for this object"""

    if obj is not None:
        
        secs=_get_docstring_sections(obj.__doc__)
        
        if 'References' in secs:
            txt= secs['References']

    print(txt)
```

It seems to work rather well in practice and print the references in rst format. in a perfect world we could also return bibtex format in option but it seems quite hard without calling some reference web API.

I used it on all estimators using function `sklearn.utils.testing.all_estimators()` :

```python
lst_all=sklearn.utils.testing.all_estimators()

for name,cl in lst_all:
    fmt='\n\n{} : {}'.format(name,cl)
    print(fmt)
    print('='*(len(fmt)-2))
    
    cite(cl)
```

 It returns a lot of nice references for the methods with some notable empty approaches actually due to documentation error (References in sklearn.svm.SVR for instance in not a proper section). It also work on any python object using the numpy docstring format with reference section so I find it quite practical.

Are you interested by this function in a PR maybe in sklearn.utils ?



user: rflamary
issue_16538: state: open
title: LinearSVC multiclass is suboptimal compared to voting
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Description
LinearSVC with multiclass data seems to do worse than creating multiple binary classifiers and voting using `decision_function()`. Since the `multiclass` default is `ovr`, this is probably not expected behavior.


#### Steps/Code to Reproduce
An [MWE](https://gist.github.com/abhishek-ghose/ff52ac298c82b70c6562bf7a29f3ef76) that demonstrates custom fitting of multiple `LinearSVC`and voting. This uses a toy dataset, where the data for a class is a vertical stripe. The code snippet generates the dataset.


The code tries the `LinearSVC()` as-is and compares it to a *custom* implementation - shown by functions `custom_ovr_fit()` and `custom_ovr_predict()`. 

#### Expected Results
There should be minor differences between the results of the two approaches.

#### Actual Results

I see something like the following on my machine, but results might vary across machines (I've tried, this is possibly because of different versions) - however the difference in the two approaches is always substantial (its `76.31%` below, but on other machines the gap closes to `~50%`.):

```
========Comparison summary========
	For c=0.25000, scikit F1=0.47, custom F1=0.83, pct. improve. over scikit = 76.94%
	For c=0.50000, scikit F1=0.48, custom F1=0.84, pct. improve. over scikit = 76.07%
	For c=1.00000, scikit F1=0.49, custom F1=0.87, pct. improve. over scikit = 76.31%
	For c=2.00000, scikit F1=0.49, custom F1=0.72, pct. improve. over scikit = 46.08%
	For c=4.00000, scikit F1=0.41, custom F1=0.53, pct. improve. over scikit = 29.29%

	**scikit: best param=1.00000, score=0.49
	**custom: best param=1.00000, score=0.87
	**pct. improve. over scikit's best score = 76.31%
```

I am using the `F1-macro` score. Not only are the values for a given `C` often not close, but the best score, across the parameter search space - represented by `pct. improve. over scikit's best score` - is drastically different.

As one might expect, this is dataset dependent. For ex on the `digits` dataset I did not see much of difference, but on the `wine` dataset, I see a gap of `~11%`. 


#### Versions
I don't have `imblearn` installed (please let me know if this is necessary) - here's the rest of the info.
```
Windows-10-10.0.17763-SP0
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]
NumPy 1.17.1
SciPy 1.1.0
Scikit-Learn 0.19.1
```
user: abhishek-ghose
issue_16536: state: open
title: control of redundant feature generation in make_classification
body: #### Describe the workflow you want to enable
To be able to limit the number of informative features which are included in the creation of the redundant features. Currently, each redundant feature is created using a random vector of coefficients for all informative features. This is not realistic for some [many?] simulation scenarios.

#### Describe your proposed solution
Add a function parameter to determine the number of informative features to use.
For each redundant feature, randomly select this number of informative features and generate random coefficients for each.

Make this optional, with default behaviour as now.
user: arc12
issue_16532: state: open
title: make_classification (samples_generator.py) explains itself
body: Noted prior issues: #10682 #10681

If this is thought acceptable, I'd be happy to make a PR for either the proposed solution or the make_regession-like alternative.

#### Describe the workflow you want to enable
I would find it useful if the structure of the generated dataset were disclosed by this function, including the combination vectors used to generate the redundant features. (and whether or not shuffling employed). This would make it easier to sanity-check e.g. feature selection procedures against expectations.

#### Describe your proposed solution
This could be as simple as a parameter to enabling print() of what is being done, in summary. This is a bit old-school, but Pipeline has a `verbose` param for the same purpose. I note a "-1" for earlier suggestion of adding a param, though.

#### Describe alternatives you've considered, if relevant
Adding a parameter similar to above but with the effect of adding a third item to the return tuple, to contain a list of codes which indicate the column origin - e.g. ["I1", "DI0", "I0", "U0"] for two informative, one duplicated (of I0) and one useless. Variable length return tuples seems bad style tho...
I note that this is how make_regression() works.

Rejected: creating a class and writing a workaround to retain current function in API.
user: arc12
issue_16517: state: open
title: Implement Extended Isolation Forest
body: #### Describe the workflow you want to enable

In the context of anomaly detection, the isolation forest algorithm has a bias making data points' anomaly scores lower than what they actually should be. This problem arises for areas in the space which are axis-aligned with a cluster. Imagine a point very far from a cluster, the basic Isolation Forest algorithm may assign it a lower anomaly score only because the point is axis aligned with the cluster. This does lead to false negative in my application.

To overcome this bias, [Hariri et al](https://arxiv.org/abs/1811.02141) proposed the Extended Isolation Forest algorithm. While the normal Isolation Forests algorithm randomly chooses a feature and a threshold value to split the points, the extended version uses a random hyperplane to do so. Those random hyperplanes, as they might not be axis-aligned, remove the bias caused by the standard algorithm. In the end, the standard algorithm becomes a subset of the extended version one, but using only axis-aligned hyperplanes.

Please have a look to the original paper, it explains the problem very well.

#### Describe your proposed solution

I had a look to the Isolation Forest code, in my humble opinion, the simplest solution might be to add an argument to the `IsolationForest` class constructor to choose how the samples should be split into two. Maybe something like an `extended` argument.

This would basically modify the `splitter` argument passed to the underlying base_estimator (an ExtraTreeRegressor instance). We could add a "random_hyperplane" splitting mode, which requires implementing a new `Splitter` class.

Globally, this is not a lot of changes but adding the new splitter class. I can work on an implementation if we agree this is a useful addition.

#### Describe alternatives you've considered, if relevant

Maybe the Extended Isolation Forest algorithm should be a distinct class, but I doubt it is worth it.

user: groud
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_16512: state: open
title: problem with nearest neighbor indices in locally_linear_embedding
body: In the function `barycenter_kneighbors_graph`, the line `ind = knn.kneighbors(X, return_distance=False)[:, 1:]` is supposed to remove the point itself from its nearest neighbors. In some cases it does not remove the point itself but another one.
Here is an example that demonstrates that:
```python
>>> import numpy as np
>>> from sklearn.neighbors import NearestNeighbors
>>> X = np.array([[0,1],[0,1],[-1,2],[5,7],[-3,2],[9,9]])
>>> n = X.shape[0]
>>> knn = NearestNeighbors(algorithm='kd_tree').fit(X)
>>> neighbors = knn.kneighbors(X, n_neighbors=3, return_distance=False)
>>> x = np.arange(n)
>>> neighbors
array([[0, 1, 2],
       [0, 1, 2],
       [2, 0, 1],
       [3, 5, 1],
       [4, 2, 1],
       [5, 3, 1]])
>>> neighbors[:,1:]
array([[1, 2],
       [1, 2],
       [0, 1],
       [5, 1],
       [2, 1],
       [3, 1]])
```
We can see that the first nearest neighbor of 1 was 0 so 1 itself is kept in the list of indices. This part fails when there are duplicate points.
In LLE, we try to approximate a point by using its neighbors, so I feel that having a non-zero value in the index of the point should be a problem.
I have solved it in my own way, but the outputs do not seem to differ, so I'm not sure whether it really is a problem.
Here's my solution:

```python
>>> mask = neighbors != x[:, np.newaxis] # find the position of the point in its KNN
>>> self_found = np.sum(~mask, axis=1) == 1 # check whether it is part of those neighbors
>>> mask[:, -1] &= self_found # remove the last row if the point was not part of its KNN
>>> neighbors[mask].reshape((n, -1)) # get the new nearest neighbors
array([[1, 2],
       [0, 2],
       [0, 1],
       [5, 1],
       [2, 1],
       [3, 1]])
```
I would like to know if it is bug worth fixing. Thank you.
user: ghost
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_16511: state: open
title: OneHotEncoder - improve flexibility of "drop" and "handle_unknown"
body: #### Describe the workflow you want to enable
There are two points here, which are somewhat related.

1. List value supplied to "drop" should be permitted to skip dropping on chosen input columns. Currently there is validation which checks that the values in the list are found categories, and this prevents selective dropping.

2. It should be possible to use both "drop" and "handle_unknown". At present a ValueError is thrown in _validate_keywords. I feel this is too strong and should be moderated to a warning (at best). Use case: drop the majority category and de-facto impute unknown categories as the majority.

#### Describe your proposed solution
For 1: allow the value None to be used in the list, to indicate no category to be dropped from the associated input column.

For 2: mutate to a warning

user: arc12
issue_16495: state: open
title: Pass additional parameters to `inverse_transform` and `transform` in pipeline.
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable

I would like to pass additional arguments to `inverse_transform` from pipeline same as you do for `fit` or `fit_transform`  (`fit(X, y=None, **fit_params))`. I think this should be consistent throughout the other functions. 

My case is that I am building a custom transformer class for [stationarizing](https://otexts.com/fpp2/stationarity.html) features. Once you apply a n order differencing, you lose the first n samples where they are replaced with nan values to keep the dataset shape consistent in the pipeline. So in inverse, I would like to pass the n rows that replaces the nan value and apply the inverse correctly.

If recording the n entries in the transformer instance is the solution, then I can't record the n entries in the `fit` function, it should be in the `transform` and if I record it in the `transform` I wouldn't be able to use the `transform` for another batch until I inverse the previous batch otherwise n entries will be overrode.

#### Describe your proposed solution

Apply the same logic of `fit_param` in `fit` function for `inverse_transform` and `transform` to keep the API consistent.

#### Describe alternatives you've considered, if relevant

#### Additional context

example of stationary by applying first order differencing:

```
x  = 2, 5, 9, 6, 12 
x_stationary = nan, 2, 4,-3, 6
# inverse x without first entry
x_inverse = nan, 2, 6, 3, 9
```
user: uzi0espil
issue_16479: state: open
title: Decision Tree probabilities with balanced class weight
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
I'm not sure if this is a bug or expected behaviour, but it's something that tripped me up and I thought I'd ask about it. The issue comes from fitting a decision tree model with `class_weight = 'balanced'`.

#### Steps/Code to Reproduce

```
import pandas as pd
import numpy as np
from sklearn import tree
from sklearn.datasets import load_breast_cancer

# Load data
data = load_breast_cancer()
X = data.data
y = data.target

# Build Decision Tree
dt = tree.DecisionTreeClassifier(
    max_depth = 1,
    class_weight = 'balanced',
    random_state = 0
).fit(
    X = X,
    y = y
)

# Output decision tree nodes and probabilities on data
node = dt.apply(X)
output_prob = dt.predict_proba(X)[:, 1]
results = pd.DataFrame({'y': y, 'node': node, 'output_prob': output_prob})

# Compare output node probabilities with calculated probabilities
def f(x):
    d = {}
    d['output_prob'] = x['output_prob'].max()
    d['calc_prob'] = sum(x['y']) / x['y'].count()
    return pd.Series(d)

comparison = results.groupby('node').apply(f)
```

#### Expected Results
I would expect that the probabilities output from `predict_proba` to match the calculated probabilities. 

#### Actual Results
The output from `predict_proba` does not match the calculated probabilities. I suspect what's happening is that the probabilities being output are calculated from the input data _after_ it's been balanced, and not from the input data itself. 

Even though the classes have been balanced during the building of the model, I wouldn't expect these balanced weightings to be used in the probabilities being generated, although maybe that's a problem with my expectations 😄. This showed up when I used the probabilities to calculate the expected versus actual target rate.

If this _is_ expected behaviour, then maybe it would be useful to have a parameter in the `predict_proba` function to allow someone to choose to output the probabilities either pre or post balancing?

#### Versions
System:
    python: 3.7.3 (default, Mar 27 2019, 22:11:17)  [GCC 7.3.0]
executable: /home/ec2-user/anaconda3/envs/test/bin/python
   machine: Linux-4.14.165-102.185.amzn1.x86_64-x86_64-with-glibc2.10

Python deps:
       pip: 19.3.1
setuptools: 42.0.2.post20191201
   sklearn: 0.21.3
     numpy: 1.17.3
     scipy: 1.3.2
    Cython: 0.29.14
    pandas: 0.25.3
user: mdpead
issue_16473: state: open
title: Support of passthrough=True in Stacking when X a dataframe of mixed type
body: In `StackingClassifier` and `StackingRegressor`, we added an option `passthrough=True/False` which will concatenate the internal predictions of the first layer model with the original dataset. While everything is going if `X` is only numerical, things start to be complicated when we are dealing with mixed types and dataframe.

Let's illustrate the issue by tacking the Titanic dataset:

## Workflow

### Some imports

```python
import numpy as np
from pandas.api.types import CategoricalDtype

from sklearn.base import clone
from sklearn.compose import make_column_selector as selector
from sklearn.compose import make_column_transformer
from sklearn.datasets import fetch_openml
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.linear_model import RidgeClassifierCV
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import OrdinalEncoder

np.random.seed(0)
```

### Load the dataset

```python
X, y = fetch_openml("titanic", version=1, as_frame=True, return_X_y=True)
subset_feature = ['embarked', 'sex', 'pclass', 'age', 'fare']
X = X[subset_feature]

X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y)
```

### Define first-layer learners

Since we are dealing with mixed-types, the first-layer learners need to define some preprocessor.
Here, we will stack a linear model and a tree-based model and then we defined 2 types of preprocessors.

```python
gradient_based_processor = make_column_transformer(
    (make_pipeline(StandardScaler(), SimpleImputer(strategy="median")),
     selector(dtype_exclude=CategoricalDtype)),
    (make_pipeline(SimpleImputer(strategy='constant', fill_value='missing'),
                   OneHotEncoder(handle_unknown='ignore')),
     selector(dtype_include=CategoricalDtype))
)

categories = [
    X[col].dtype.categories.tolist() + ["missing"] if X[col].isnull().any()
    else X[col].dtype.categories.tolist()
    for col in selector(dtype_include=CategoricalDtype)(X)
]

tree_based_processor = make_column_transformer(
    (make_pipeline(SimpleImputer(strategy='constant', fill_value='missing'),
                   OrdinalEncoder(categories=categories)),
     selector(dtype_include=CategoricalDtype))
)

rf = make_pipeline(clone(tree_based_processor), RandomForestClassifier())
lr = make_pipeline(clone(gradient_based_processor), LogisticRegression())

# sanity check
print(lr.fit(X_train, y_train).score(X_test, y_test))
print(rf.fit(X_train, y_train).score(X_test, y_test))
```
```
0.8048780487804879
0.8048780487804879
```

Up to now, we have a standard learner. Let's try to stack them

### Only stack the predictions of the first-layer models

Let's use some stacking. By default, we will stack the predictions of the first-layer models

```python
numerical_ridge = RidgeClassifierCV()
model = StackingClassifier(
    estimators=[("lr", lr), ("rf", rf)],
    final_estimator=numerical_ridge,
    passthrough=False
)
print(model.fit(X_train, y_train).score(X_test, y_test))
```
```
0.8048780487804879
```

We don't have any issues since the predictions are only numerical. The issue starts if we want to concatenate the original `X`

### Concatenate `X` with the predictions of the first-layer models

If we try directly to pass through `X` we get an issue since the ridge classifier did not encode the data in `X`:

```python
model = StackingClassifier(
    estimators=[("lr", lr), ("rf", rf)],
    final_estimator=numerical_ridge,
    passthrough=True
)
try:
    print(model.fit(X_train, y_train).score(X_test, y_test))
except Exception as e:
    print(e)
```
```
could not convert string to float: 'S'
```

Thus, one has to create a pipeline to deal with the data:

```python
ridge = make_pipeline(clone(gradient_based_processor), RidgeClassifierCV())
ridge.named_steps["columntransformer"].set_params(remainder="passthrough")

model = StackingClassifier(
    estimators=[("lr", lr), ("rf", rf)],
    final_estimator=ridge,
    passthrough=True
)
try:
    print(model.fit(X_train, y_train).score(X_test, y_test))
except Exception as e:
    print(e)
```
```
make_column_selector can only be applied to pandas dataframes
```

So here we actually have the following issue:

* the concatenation does not preserve dataframe. This could be solved by stacking all the data to `X` (which should manage without importing `pandas`). However, it means that we might want to generate some **feature names** for the predictions generated by the first-layers. So this is my first question, what shall we do?
* if the above issue can be fixed, then we start to have another issue. When creating the pipeline, the user would need to specifically state to let pass through the remainder in the final estimator. Otherwise, the prediction columns will be dropped. So, we might want to interfere with the column transformer of the last steps to be sure to pass through the predictions. But API I don't know what is the best way to do so?

ping @jnothman @qinhanmin2014 @thomasjpfan 

Sorry for the long narration.
user: glemaitre
issue_16470: state: open
title: Make _binary_clf_curve a "public" method?
body: Currently `sklearn.metrics.ranking._binary_clf_curve` is (the way I understand the underscore) an internal API method.

Whenever there is a need to work with a different tradeoff than precision / recall or roc or when you need custom metrics for all thresholds, this method is a perfect fit, and the underscore in front of it makes me wonder if I can be confident it will not change in future versions :-)  

I need to compute for instance (FP+TN)/(TN+FN+FP) at different thresholds and other use cases could be e.g. [these](https://stackoverflow.com/questions/32589447/using-sklearn-internal-function))

I think making this method part of the public API would be beneficial for the community.

p.s.
Tensorflow used to have e.g. `tensorflow.contrib.metrics.python.ops.metric_ops.precision_recall_at_equal_thresholds` now they have https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/metrics_impl.py#L1792 etc.
user: Kornel
issue_16469: state: open
title: Deprecate _estimator_type, replace by estimator tag
body: I think we should deprecate ``_estimator_type`` which predates ``estimator_tags``. It's private but we should probably still deprecate it as it can lead to weird errors downstream. If we change it and downstream is not updating it, it would lead to classifiers using KFold cross-validation, which can be really bad if the target is sorted.
user: amueller
issue_16463: state: open
title: added LinearOperator support in safe_sparse_dot()
body: 
#### Reference Issues/PRs
Fixes #16461.


user: PavelStishenko
labels: Needs Decision: Requires decision
issue_16461: state: open
title: Add support of LinearOperator in randomized_svd
body: #### Describe the workflow you want to enable
It would be nice if `randomized_svd()` worked with descendants of `scipy.sparse.linalg.LinearOperator`. 
```
import numpy as np
from scipy.sparse.linalg import aslinearoperator
from sklearn.utils.extmath import randomized_svd

A = np.random.random((4,4))
op = aslinearoperator(A)
randomized_svd(op, 2)
```
This code currently raises exception from `line 134, in safe_sparse_dot`:
```
AttributeError: 'MatrixLinearOperator' object has no attribute 'ndim'
```

#### Describe your proposed solution
Fixed version of `safe_sparse_dot`:
```
def safe_sparse_dot(a, b, dense_output=False):
    """Dot product that handle the sparse matrix case correctly

    Parameters
    ----------
    a : array or sparse matrix
    b : array or sparse matrix
    dense_output : boolean, (default=False)
        When False, ``a`` and ``b`` both being sparse will yield sparse output.
        When True, output will always be a dense array.

    Returns
    -------
    dot_product : array or sparse matrix
        sparse if ``a`` and ``b`` are sparse and ``dense_output=False``.
    """
    if len(a.shape) > 2 or len(b.shape) > 2:
        if sparse.issparse(a):
            # sparse is always 2D. Implies b is 3D+
            # [i, j] @ [k, ..., l, m, n] -> [i, k, ..., l, n]
            b_ = np.rollaxis(b, -2)
            b_2d = b_.reshape((b.shape[-2], -1))
            ret = a @ b_2d
            ret = ret.reshape(a.shape[0], *b_.shape[1:])
        elif sparse.issparse(b):
            # sparse is always 2D. Implies a is 3D+
            # [k, ..., l, m] @ [i, j] -> [k, ..., l, j]
            a_2d = a.reshape(-1, a.shape[-1])
            ret = a_2d @ b
            ret = ret.reshape(*a.shape[:-1], b.shape[1])
        else:
            ret = np.dot(a, b)
    else:
        if isinstance(b, LinearOperator) and not isinstance(a, LinearOperator):
            ret = (b.T @ a.T).T
        else:
            ret = a @ b

    if (sparse.issparse(a) and sparse.issparse(b)
            and dense_output and hasattr(ret, "toarray")):
        return ret.toarray()
    return ret
```

Key changes are the following:
1) Use len(a.shape) instead of a.ndim
2) Use transposed multiplication to use LinearOperator's dot product routines

user: PavelStishenko
issue_16457: state: open
title: mutual_info_regression is missing a factor?
body: ```
from pylab import *

from sklearn.feature_selection import mutual_info_regression

mi = dict()
for n in np.logspace(2, 5, 10):
    n = int(n)
    x = np.random.randn(n)[:, None]
    mi[n] = mutual_info_regression(x, x.squeeze(), n_neighbors=5).item()
mi = pd.Series(mi)
mi.name = 'mi'
mi.index.names = ['n']
mi = mi.reset_index()

figure()
plot(mi.n, mi.mi, 'o')
xscale('log')
grid()
figure()
plot(mi.n, mi.mi - np.log(mi.n), 'o')
xscale('log')
ylim(0, ylim()[1] * 2)
grid()
```

![image](https://user-images.githubusercontent.com/223276/74650392-873c7380-5179-11ea-95d4-34d8c004bd08.png)

user: cottrell
issue_16443: state: open
title: 0.22.1 test_stacking ends with Bus error on ARMv7 arch (Python 3.5)
body: #### Describe the bug
Running standard test suite for Scikit-learn 0.22.1 on ARMv7 architecture for Ubuntu 16.04 (Python 3.5) results in fatal bus error.

```
# scikit_learn-0.22.1-cp35-cp35m-linux_armv7l.whl
> pytest -v /usr/local/lib/python3.5/dist-packages/sklearn
========================= test session starts =========================
platform linux -- Python 3.5.2, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /usr/local/lib/python3.5/dist-packages/sklearn
collected 14072 items / 3 skipped / 14069 selected

...

../../../../usr/local/lib/python3.5/dist-packages/sklearn/ensemble/tests/test_stacking.py::test_check_estimators_stacking_estimator[StackingClassifier] PASSED                                              [ 11%]
../../../../usr/local/lib/python3.5/dist-packages/sklearn/ensemble/tests/test_stacking.py::test_check_estimators_stacking_estimator[StackingRegressor] Fatal Python error: Bus error

Thread 0xa7fff470 (most recent call first):
  File "/usr/lib/python3.5/threading.py", line 293 in wait
  File "/usr/local/lib/python3.5/dist-packages/joblib/externals/loky/backend/queues.py", line 138 in _feed
  File "/usr/lib/python3.5/threading.py", line 862 in run
  File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

Thread 0xa892f470 (most recent call first):
  File "/usr/lib/python3.5/selectors.py", line 376 in select
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 911 in wait
  File "/usr/local/lib/python3.5/dist-packages/joblib/externals/loky/process_executor.py", line 615 in _queue_management_worker
  File "/usr/lib/python3.5/threading.py", line 862 in run
  File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

Current thread 0xb6ff6300 (most recent call first):
  File "/usr/local/lib/python3.5/dist-packages/sklearn/tree/_classes.py", line 367 in fit
  File "/usr/local/lib/python3.5/dist-packages/sklearn/tree/_classes.py", line 1225 in fit
  File "/usr/local/lib/python3.5/dist-packages/sklearn/ensemble/_base.py", line 36 in _parallel_fit_estimator
  File "/usr/local/lib/python3.5/dist-packages/joblib/parallel.py", line 256 in <listcomp>
  File "/usr/local/lib/python3.5/dist-packages/joblib/parallel.py", line 256 in __call__
  File "/usr/local/lib/python3.5/dist-packages/joblib/_parallel_backends.py", line 590 in __init__
  File "/usr/local/lib/python3.5/dist-packages/joblib/_parallel_backends.py", line 209 in apply_async
  File "/usr/local/lib/python3.5/dist-packages/joblib/parallel.py", line 754 in _dispatch
  File "/usr/local/lib/python3.5/dist-packages/joblib/parallel.py", line 835 in dispatch_one_batch
  File "/usr/local/lib/python3.5/dist-packages/joblib/parallel.py", line 1007 in __call__
  File "/usr/local/lib/python3.5/dist-packages/sklearn/ensemble/_stacking.py", line 141 in fit
  File "/usr/local/lib/python3.5/dist-packages/sklearn/ensemble/_stacking.py", line 643 in fit
  File "/usr/local/lib/python3.5/dist-packages/sklearn/utils/estimator_checks.py", line 2201 in check_regressors_train
  File "/usr/local/lib/python3.5/dist-packages/sklearn/utils/_testing.py", line 327 in wrapper
  File "/usr/local/lib/python3.5/dist-packages/sklearn/utils/estimator_checks.py", line 427 in check_estimator
  File "/usr/local/lib/python3.5/dist-packages/sklearn/ensemble/tests/test_stacking.py", line 382 in test_check_estimators_stacking_estimator
  File "/usr/local/lib/python3.5/dist-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
  File "/usr/local/lib/python3.5/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.5/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.5/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.5/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.5/dist-packages/_pytest/python.py", line 1445 in runtest
  File "/usr/local/lib/python3.5/dist-packages/_pytest/runner.py", line 134 in pytest_runtest_call
  File "/usr/local/lib/python3.5/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.5/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.5/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.5/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.5/dist-packages/_pytest/runner.py", line 210 in <lambda>
  File "/usr/local/lib/python3.5/dist-packages/_pytest/runner.py", line 237 in from_call
  File "/usr/local/lib/python3.5/dist-packages/_pytest/runner.py", line 210 in call_runtest_hook
  File "/usr/local/lib/python3.5/dist-packages/_pytest/runner.py", line 185 in call_and_report
  File "/usr/local/lib/python3.5/dist-packages/_pytest/runner.py", line 99 in runtestprotocol
  File "/usr/local/lib/python3.5/dist-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol
  File "/usr/local/lib/python3.5/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.5/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.5/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.5/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.5/dist-packages/_pytest/main.py", line 271 in pytest_runtestloop
  File "/usr/local/lib/python3.5/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.5/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.5/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.5/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.5/dist-packages/_pytest/main.py", line 247 in _main
  File "/usr/local/lib/python3.5/dist-packages/_pytest/main.py", line 197 in wrap_session
  File "/usr/local/lib/python3.5/dist-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/usr/local/lib/python3.5/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.5/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.5/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.5/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.5/dist-packages/_pytest/config/__init__.py", line 93 in main
  File "/usr/local/bin/pytest", line 8 in <module>
Bus error
```

#### Versions
System:
    python: 3.5.2 (default, Oct  8 2019, 13:06:37)  [GCC 5.4.0 20160609]
   machine: Linux-3.8.13.30-armv7l-with-Ubuntu-16.04-xenial
executable: /usr/bin/python3

Python dependencies:
matplotlib: None
     scipy: 1.4.1 (test suite run without failures)
     numpy: 1.18.1 (test suite run without failures)
       pip: 20.0.2
   sklearn: 0.22.1
setuptools: 45.2.0
    pandas: None
    Cython: None (was installed latest from pip and deleted after building scikit-learn)
    joblib: 0.14.1

Built with OpenMP: True
user: h6197627
labels: arch:arm: ARM related issues
issue_16439: state: open
title: [WIP] PERF Parallelize W/H updates of NMF with OpenMP
body: Continuation of #6641

Here's a benchmark of the speedup using 2 threads. When the number of components is large enough the speed-up approaches 2x (maybe a bit better than 2x because I switched to a blas `dot` instead of a manual loop).
![image](https://user-images.githubusercontent.com/34657725/74445723-24946080-4e77-11ea-9924-f3bf6ce36fdf.png)

Since it now uses a BLAS function inside an OpenMP loop, we first need to prevent oversubscription with threadpoolctl before we can merge this pr.
user: jeremiedbb
issue_16433: state: open
title: LinearDiscriminantAnalysis.predict uses all dimensions, ignoring n_components
body: #### Describe the bug

When using LinearDiscriminantAnalysis, the result of predict seems unchanged if we change the parameter n_components, even for problems where it seems obvious that 2 components separate the points better than 1 component. It seems that to predict a label, it is always using more components than n_components.

#### Steps/Code to Reproduce


Example:
```python
import matplotlib.pyplot as plt
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.datasets import load_wine

wine = load_wine()
X = wine.data
y = wine.target

clf1 = LinearDiscriminantAnalysis(n_components=1, solver='svd')
clf1.fit(X, y)
Xt = clf1.transform(X)
y_pred = clf1.predict(X)

plt.figure()
plt.scatter(Xt[:, 0], Xt[:, 0], c=y)
plt.figure()
plt.scatter(Xt[:, 0], Xt[:, 0], c=y_pred)

plt.show()

```

#### Expected and actual results

Wine dataset is not separable with a single lda component. However, predict seems to give the right labels. 

I would have expected the diagonal in the second plot to be divided cleanly in three intervals, one for each class.

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
import imblearn; print("Imbalanced-Learn", imblearn.__version__)
-->

>>> import sklearn; sklearn.show_versions()

System:
    python: 3.7.4 (default, Aug 13 2019, 20:35:49)  [GCC 7.3.0]
executable: /home/user/.miniconda3/envs/torch/bin/python
   machine: Linux-4.19.0-6-amd64-x86_64-with-debian-10.2

Python dependencies:
       pip: 19.2.2
setuptools: 41.0.1
   sklearn: 0.22.1
     numpy: 1.16.4
     scipy: 1.3.1
    Cython: None
    pandas: 0.25.1
matplotlib: 3.0.3
    joblib: 0.13.2

Built with OpenMP: True

>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.16.4
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.3.1
>>> import sklearn; print("Scikit-Learn", sklearn.__version__)
Scikit-Learn 0.22.1


<!-- Thanks for contributing! -->

user: fjhheras
issue_16429: state: open
title: HistGradientBoostingClassifier slow in prediction mode
body: While HistGradientBoostingClassifier is 100 faster than GradientBoostingClassifier when fitting the model, I found it to be very slow in case of predicting the class probabilities, in my case about 100 times slower :-(

For example: 
GradientBoostingClassifier: 3.2 min for training for 1 million examples. 32 ms for 1000 predictions.
HistGradientBoostingClassifier: 7s for training. 1s for 1000 predictions
user: SebastianBr
issue_16419: state: open
title: X_norm_squared is not chunked on pairwise_distances_chunked
body: #### Describe the bug

I believe there is a bug on sklearn.metrics.pairwise.pairwise_distances_chunked(). If `X` gets chunked, and you pass `X_norm_squared` in `kwds`, this would throw a `ValueError: Incompatible dimensions for X and X_norm_squared`  in `euclidean_distances()` because `X_norm_squared ` wasn't chunked in the same way as `X`.

#### Steps/Code to Reproduce

`X.shape = (1378500, 100)`
`kernels.shape  = (100, 100)`
`XX.shape = (1378500, 1)`

`X.dtype = dtype('float64')`
 
```python
best_labels, best_distances = \
    pairwise_distances_argmin_min(
        X=X,
        Y=kernels,
        metric_kwargs={'squared': True, 'X_norm_squared': XX})
```

#### Expected Results

No error is thrown.

#### Actual Results

line 273, in euclidean_distances
    "Incompatible dimensions for X and X_norm_squared")
ValueError: Incompatible dimensions for X and X_norm_squared

#### Versions

System:
    python: 3.7.6 (default, Jan  8 2020, 19:59:22)  [GCC 7.3.0]
executable: $HOME/local/bin/anaconda/anaconda3/envs/tuh_pyedflib_test/bin/python
   machine: Linux-5.0.0-37-generic-x86_64-with-debian-buster-sid

Python dependencies:
       pip: 20.0.2
setuptools: 45.1.0.post20200119
   sklearn: 0.22.1
     numpy: 1.18.1
     scipy: 1.4.1
    Cython: None
    pandas: None
matplotlib: 3.1.1
    joblib: 0.14.1

Built with OpenMP: True

<!-- Thanks for contributing! -->
user: chmendoza
issue_16418: state: open
title: [WIP] Example generating various distance metrics on RandomTreesEmbedding and 
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Ref PR: #1288

#### What does this implement/fix? Explain your changes.
- adds an example of generating distance metrics from RandomTreesEmbedding and uses it on unsupervised task. 

The distance metrics used are: 
1) the depth of the nearest common ancestor of the leaf nodes sample i and j land into (take the reciprocal to make it a distance)
2) the length of the shortest path of the leaf nodes sample i and j land into
3) the probability of sample i and j landing into the same lead node(take 1 minus the value to make it a distance)

#### Any other comments?
The plot now looks like this: 
![image](https://user-images.githubusercontent.com/55062863/74109846-23042900-4b55-11ea-960b-cac5d680474a.png)


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: ShumingXu
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_16399: state: open
title: Improve check_classification_targets to warn users about large number of classes and avoid redundant computation
body: - `check_classification_targets` should warn the user if the number of unique classes is larger than say 50% of  `n_samples`: the user is likely feeding bad targets which can be highly problematic when the size of the model depends linearly (or more) on the number of classes (e.g. Gradient Boosting models, OneVsOne SVM, ...).

- `check_classification_targets` which calls `type_of_target` triggers a call to `_assert_all_finite` and `np.unique(y)` which are redundant with checks done elsewhere (e.g. in `check_X_y`)

- `check_classification_targets` should probably be refactored to return the array of unique classes (to be used for the `classes_` attribute of the classifier) to avoid redundant computation.

- `check_classification_targets` should probably be used more consistently for any classifier maybe by being used internally in `check_X_y` or in `_validate_data` instead of being called manually on a case by case basis.

And we probably need a common test to check the above (e.g. all classifier should raise the warning).
user: ogrisel
issue_16398: state: open
title: Make RidgeCV, LogisticRegressionCV, ... warn when found optimal regularization parameter lies at the boundary of the range
body: I think the `RidgeCV().fit(X_train, y_train)` should warn the user if the found value for `alpha_` is either `alphas.min()` or `alphas.max()`. E.g.

```
StatisticalWarning: the optimal value for the regularization parameter 'alpha' was 0.01 which lies at a boundary of the explored range (between 0.01 and 1.). Consider setting the 'alphas' parameter to explore a wider range. 
```

We could add a new `boundary_warning=True` constructor parameter to make it possible to silence the warning.

BTW, the default ranges could probably be extended between 1e-6 and 1e6 with 13 levels on the logspace whenever it is cheap to do so (e.g. for RidgeCV whos current default range much too narrow `[0.1, 1., 10.]`).
user: ogrisel
issue_16393: state: open
title: Estimator data source object
body: Hi! The pipeline instances is very interesting because it uses a sintaxis very similar to the functional programming frameworks, such as pyspark and apache beam. However, I miss a "DataSource" object which dealed with the logic required to load the data beyond passing it as a list of values. For instance, it could be able to connect to a database to load the data for fitting and for predicting. 

```
Pipeline([MongoDataSource("127.0.0.1:8087", query),
          Normalizer(),
          SVC()]
          ).fit()
```
I have tried to build one of those objects by my own using an Estimator structure, however, I saw that it does not fit your conventions (https://scikit-learn.org/stable/developers/develop.html). That is why I think that is needed a new object

What do you think? Thank you!
user: DelgadoPanadero
issue_16383: state: open
title: Add conformal prediction functionality
body: Understanding the quality of a prediction is a very important use case which is not supported by scikit-learn. Quartile Regression is about as close as it gets. The formal solution to this is conformal predictions. 

There is already a library built on scikit-learn which does this.

https://github.com/donlnz/nonconformist/

Is was written by an expert in the field but is no longer maintained. It might be a good idea to bring this inside the scikit-learn code base while it is still simple to do.
user: Gitman-code
issue_16381: state: open
title: More memory efficient LinearClassifierMixin.predict
body: I'm running out of memory with LogisticRegression.predict on a dataset with,
 - n_samples = 1M, n_feature=8k, n_classes=800

This happens as `LinearClassifierMixin.predict`, needs to compute the decision function for all classes which involves a `(n_samples, n_features) x (n_features, n_classes)` multiplication, resulting in a `(n_samples, n_classes)` dense matrix (in my case ~6 GB).

Batching in n_samples is a solution, but it would have been nice to have a standard way of doing that, either as a util function that would be used to decorate e.g. `batch_apply(LinearClassifierMixin.predict)(X)` or a fit param `LinearClassifierMixin.predict(X, batch_size=1000)`. 

This is somewhat related to parallizing predict in general (since there batching also happens). Can't find the corresponding issue right now.

This particular case is made worse by the fact that `LogisticRegression.coef_` is not enforced to be float32 for float32 sparse training input (with the liblinear solver) related https://github.com/scikit-learn/scikit-learn/issues/8769.
user: rth
issue_16371: state: open
title: No hyperparameter optimization example/tutorial incorporating warmstart
body: #### Description

Currently, there is no example or demonstration of a hyperparameter optimization function that uses warmstart (GridSearchCV and RandomSearchCV ignore warmstart) and tunes multiple classifier types in the same function.  For comparisons between algorithm performance on multiple datasets, a common experiment for tasks such as benchmarking, such a function would be especially useful and reduce computational time.

#### Enhancement

- I will write a function that, given a classifier and a dictionary of two hyperparameters, finds the optimal pair of hyperparameter values using grid search.  In this specific example, max_features and n_estimators will be the parameters being tuned.  

- The function will implement warmstart for searching through increasing numbers of estimators in the ensemble.

- A function will be included that visualizes the performance of each parameter value pair in a heatmap to show more detail on how the optimal parameters were determined.
user: cshih14
issue_16370: state: open
title: Lack of comparisons between regression forest split criteria
body: #### Describe the issue linked to the documentation
There are several types of split criteria (parameter: criterion) to choose from when performing regression using random forest in scikit-learn, but there doesn't seem to be sufficient documentation comparing these split criteria.
<!--
Tell us about the confusion introduced in the documentation.
-->

#### Suggest a potential alternative/fix
An example that plots how prediction score is affected by which split criterion you use for several simulated datasets could shed light on how the split criteria compare in different situations.
<!--
Tell us how we could improve the documentation in this regard.
-->

user: morgsmss7
issue_16369: state: open
title: ENH: Conditional Density Estimation Loss - New Decision Tree Criterion
body: This PR implements conditional density estimation loss as described in T. Pospisil, and A. Lee, "RFCDE: Random Forests for Conditional Density Estimation", arXiv:1804.05753 [stat.ML], 2018. The key advantages of adding the CDE Loss criterion into sklearn is for making it easy for users to incorporate this algorithm in scikit-learn pipelines and serialize this estimator. 

Conditional Density Estimation is a useful branch of machine learning with many applications. The random forest implementation in scikit-learn is robust, in comparison to the C++ implementation provided by the authors of the above paper.

While users can easily implement their own cython class for CDE Loss, I believe this criterion is useful enough for CDE-type problems (and similar enough to the existing MSE criterion), that it should be supported within scikit-learn. 
user: vsriram11
issue_16367: state: open
title: target_names in dataset loaders can mean "unique classes" or "target columns"
body: `fetch_openml` and `load_linnerud` uses `target_names` to mean "name of target columns", which other dataset loads uses `target_names` to mean "unique classes".
user: thomasjpfan
labels: Needs Decision: Requires decision
issue_16360: state: open
title: DBSCAN output seems to be wrong in precomputed method
body: <!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
<!--
A clear and concise description of what the bug is.
-->
I'm using DBSCAN to fined clusters out of my data. but I don't have the actual data points, but I have their similarity matrix. so I used 'precomputed' method and passed (1 - similarity_matrix) to the DBSCAN clustering algorithm. but in the output labels, I found a cluster having just one data point at it. it seems not right. i set eps=0.3 in a similarity matrix with values in {[0.7-1] + 0} and mean samples min_samples=2.

as the DBSCAN algorithm works, it finds core points, which have at least min_samples data points in their eps distance and make them all a new cluster, then go to each near point and checks if it is a core point, and so on. so a cluster with 1 data point seems to be wrong.

#### Steps/Code to Reproduce
<!--
Please add a minimal example that we can reproduce the error by running the
code. Be as succinct as possible, do not depend on external data. In short, we
are going to copy-paste your code and we expect to get the same
result as you.

Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
docs = ["Help I have a bug" for i in range(1000)]
vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)
lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

```
from sklearn.cluster import DBSCAN
sim = get_similarity_matrix(text_lines)
sim[sim < 0.7] = 0
clf = DBSCAN(eps=0.3, min_samples=2, metric='precomputed')
cls = clf.fit(1 - sim).labels_
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
this **12** in Actual Results is the problem. this should not happen as I think. 
I analyzed the data and results and I think this data point should be labeled as an outlier or cluster number 5. I think cluster 5 is a better choice in my parameter space.

#### Actual Results
[ 0, -1, -1, 1, 2, -1, 3, -1, 1, -1, -1, -1, -1, -1, -1, 4, 5, 2, 1, 3, 6, 7, 8, 9, 10, 5, 2, 4, 3, 8, 1, 11, **12**,  6, 9, 1, 2, 3, 10, 8, 6, 11, 4, 9, 5, 2, 6, 4, 9, 5, -1, 2, 6, -1, 4, -1, 1, 5, 4, 5, 8, 2, 6, 3, 1, 9, 4, 10, 5, 11, 3, 4, 8, 2, 1, 6, 9, 5, 11, 2, 3, 8, 6, 4, 9, 5, 8, 3, 9, 5, -1, 4, -1, 0, 8, 10, 5, -1, 4, 2, 1, 6, -1, -1, 2, 1, 11, 4, 6, 2, 4, 1, -1, 4, 1, 2, 4, 8, 3, 5, 2, 1, 10, 0]



#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
import imblearn; print("Imbalanced-Learn", imblearn.__version__)
-->
System:
    python: 3.6.9 (default, Nov  7 2019, 10:44:02)  [GCC 8.3.0]
executable: /usr/bin/python3.6
   machine: Linux-4.15.0-76-generic-x86_64-with-Ubuntu-18.04-bionic
Python deps:
       pip: 9.0.1
setuptools: 44.0.0
   sklearn: 0.21.3
     numpy: 1.18.1
     scipy: 1.4.1
    Cython: 0.29.14
    pandas: 0.23.4

<!-- Thanks for contributing! -->

user: mohamad-qodosi
issue_16353: state: open
title: GridSearchCV and RandomizedSearchCV: option to compute a single score from all CV splits
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
I want to optimize hyperparameters for a classifier. My data has three classes, of sizes 20, 12, and 7. I want to use cross-validation with at least 10 classes, so that each estimator will have at least 35 training samples.

For each combination of hyperparameters, I want to combine the predictions for all splits, and compute a single score from all of the predictions and all of the true labels.

#### Describe your proposed solution
`GridSearchCV` and `RandomizedSearchCV` compute a score for each split, and then average them. I'd like them to have an option to collect the predictions from all splits, and compute a single score using all of them. For most metrics, this will give a different answer.

#### Describe alternatives you've considered, if relevant
One option is to not use `GridSearchCV` or `RandomizedSearchCV`. Instead I can write my own code to generate parameter combinations and CV splits, and directly call each estimator's `fit()` and `predict()` functions once for each split. 

For some metrics, it's possible to write a custom scorer such that the average of the single-split scores, weighted  using `iid=True`, will give the score I want. For example, to get balanced accuracy, the custom scorer would weight each class according to the total number samples of that class in all splits. 

I don't believe it's possible to get the F1 Macro score as the average of single-split scores. Instead, I could use `GridSearchCV` with nine custom scorers to get true positives, false positives, and false negatives for each class. After the `GridSearchCV` is fitted, I could get the single-split scores from `cv_results_`, add them across all splits, and use these sums to compute F1 Macro for each parameter combination.
#### Additional context
user: david-r-wasserman
issue_16350: state: open
title: Unable to Take Cartesian Products of Many Arrays
body: 
#### Describe the bug
When trying to run cartesian from sklearn.utils.extmath on more 32 arrays, the module crashes due to np.indices functioning rejecting the input shape in the source code.

#### Steps/Code to Reproduce
cartesian(([(1, 2), (3, 4)], [(5, 6), (7, 8)], [(9, 10), (7, 11)], [(3, 12), (13, 14)], [(9, 15), (3, 16)], [(17, 18), (3, 19)], [(1, 20), (21, 22)], [(21, 23), (3, 24)], [(25, 26), (27, 28)], [(21, 29), (3, 30)], [(9, 31), (27, 32)], [(9, 33), (1, 34)], [(21, 35), (17, 36)], [(13, 37), (7, 38)], [(25, 39), (7, 40)], [(1, 41), (27, 42)], [(21, 43), (17, 44)], [(17, 45), (3, 46)], [(25, 47), (17, 48)], [(21, 49), (17, 50)], [(5, 51), (13, 52)], [(1, 53), (7, 54)], [(25, 55), (13, 56)], [(5, 57), (7, 58)], [(9, 59), (1, 60)], [(25, 61), (5, 62)], [(3, 63), (27, 64)], [(25, 65), (7, 66)], [(1, 67), (27, 68)], [(27, 69), (7, 70)], [(21, 71), (17, 72)], [(17, 73), (5, 74)], [(9, 75), (21, 76)], [(21, 77), (13, 78)], [(25, 79), (3, 80)], [(9, 81), (25, 82)], [(9, 83), (7, 84)], [(17, 85), (27, 86)], [(5, 87), (27, 88)], [(25, 89), (13, 90)]))

Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
docs = ["Help I have a bug" for i in range(1000)]
vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)
lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

```
Sample code to reproduce the problem
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
The cartesian product of the lists.
#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->

Traceback (most recent call last):
  File "/home/hlc5v/.conda/envs/graph_env/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/hlc5v/.conda/envs/graph_env/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/hlc5v/.conda/envs/graph_env/lib/python3.6/cProfile.py", line 160, in <module>
    main()
  File "/home/hlc5v/.conda/envs/graph_env/lib/python3.6/cProfile.py", line 153, in main
    runctx(code, globs, None, options.outfile, options.sort)
  File "/home/hlc5v/.conda/envs/graph_env/lib/python3.6/cProfile.py", line 20, in runctx
    filename, sort)
  File "/home/hlc5v/.conda/envs/graph_env/lib/python3.6/profile.py", line 64, in runctx
    prof.runctx(statement, globals, locals)
  File "/home/hlc5v/.conda/envs/graph_env/lib/python3.6/cProfile.py", line 100, in runctx
    exec(cmd, globals, locals)
  File "../../DNF_Approx/GraphConvert/graphtodnf.py", line 191, in <module>
    main()
  File "../../DNF_Approx/GraphConvert/graphtodnf.py", line 176, in main
    pi, vars, weights = createPathSet(curr, infected, args.susceptible[j], t=int(args.time_steps))
  File "../../DNF_Approx/GraphConvert/graphtodnf.py", line 103, in createPathSet
    PI = cartesian_product(*pathSet)
  File "../../DNF_Approx/GraphConvert/graphtodnf.py", line 113, in cartesian_product
    arr = np.empty([len(a) for a in arrays] + [la], dtype=tuple)
ValueError: sequence too large; cannot be greater than 32

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
import imblearn; print("Imbalanced-Learn", imblearn.__version__)
-->
System:
    python: 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)]
executable: C:\Users\hcars\PycharmProjects\DNF_git\GraphForecastApprox\venv\Scripts\python.exe
   machine: Windows-10-10.0.17763-SP0
Python dependencies:
       pip: 19.0.3
setuptools: 40.8.0
   sklearn: 0.22.1
     numpy: 1.17.2
     scipy: 1.4.1
    Cython: None
    pandas: 0.25.1
matplotlib: 3.1.1
    joblib: 0.14.1
Built with OpenMP: True


<!-- Thanks for contributing! -->

user: hcars
issue_16317: state: open
title: github action to build the doc on the master branch once per week and open a new github issue whenever any new FeatureWarning / DeprecationWarning is introduced
body: This is just an idea for now but if other people find this is a good idea this could be a git first issue for devops minded contributors who would like to start to contribute to scikit-learn.

Things to check: that the github actions build duration limit is long enough for us to build the doc with all the examples without reaching the limit.
user: ogrisel
issue_16311: state: open
title: check_estimator assumes too much about 3rd-party estimators
body: I am writing an estimator that I'd like to make compatible with scikit-learn. My estimator has a parameter/field called `n_components`. Note that the purpose/meaning/intent of this parameter is different from parameters of the same name of existing scikit-learn estimators (e.g. `LinearDiscriminantAnalysis` or `LinearDiscriminantAnalysis`).

When I run `check_estimator()` on my model it fais because `check_estimator()` modifies my `n_components` field in [this piece of code](https://github.com/scikit-learn/scikit-learn/blob/0.22.X/sklearn/utils/estimator_checks.py#L971-L972)

There are plenty more places in `estimator_checks.py` where estimators' `n_components` or `n_clusters` fields are being set to specific values **without checking for estimator type**. At the very least type checks should be added to all of those cases in a manner similar to [this "ugly" piece of code](https://github.com/scikit-learn/scikit-learn/blob/0.22.X/sklearn/utils/estimator_checks.py#L485-L488).

Looks like this issue might be complementary to https://github.com/scikit-learn/scikit-learn/issues/6715 which deals with `check_estimator()` not being general enough.
user: kasparthommen
issue_16301: state: open
title: Factory-style construction of composite estimators
body: I find the API for specifying lists constituent estimators in Pipeline, FeatureUnion, ColumnTransformer, StackingClassifier, VotingClassifier, GeneralizedNB, etc. ugly. I find the `make_pipeline`, `make_column_transformer` etc interfaces helpful, but the duplication of interfaces is actually quite ugly too, and it is limiting: the moment you want to name one of your Pipeline steps explicitly, in order to make your grid search parameter space reasonably intelligible, you need to change from `make_pipeline` syntax to `Pipeline` syntax. Yuck!

I'd like to see a POC implementation and likely a [SLEP](https://scikit-learn-enhancement-proposals.readthedocs.io/en/latest/) towards creating a syntax like:

```py
est = (ColumnTransformer()
       .append(CountVectorizer(), 'body_text')
       .append(KNNImputer(), ['age'], name='impute')
       .fit(X))
```

The `append` method would progressively construct the `transformers` parameter of `est`. Note that the first use of append does not give the body_text transfomer a name, so one will be automatically generated.

Note that since we delay parameter validation until `fit`, similar syntax is already possible:
```py
est = (LogisticRegression()
       .set_params(penalty='l1')
       .fit(X, y))
```

So additional methods like `append` are just syntactic sugar for getting, amending and setting the steps/transformers parameter. But I think they would encourage named parameters instead of confusing tuples, creating more legible code.

Each of the above estimators would have methods similar to `append` or `add` (perhaps also `insert`).

PS: I have mentioned this proposal before, particularly when we ran into the inconsistent tuple ordering between `make_column_transformer` and `ColumnTransformer`.
user: jnothman
issue_16298: state: open
title: Check that zero sample weight means samples are ignored
body: An issue with an associated common check originally discussed in https://github.com/scikit-learn/scikit-learn/pull/15015 

> This is a pretty simple sample_weight test that says that a weight of 0 is equivalent to not having the samples.
> I think every failure here should be considered a bug. This is:

- [ ] SVR
- [ ] SGDRegressor
- [ ] RANSACRegressor
- [ ] OneClassSVM
- [ ] NuSVR
- [ ] MiniBatchKMeans
- [ ] LinearSVR
- [ ] LinearSVC
- [ ] LogisticRegressionCV
- [ ] KernelDensity
- [ ] KMeans
- [ ] CalibratedClassifierCV
- [ ] IsolationForest
- [ ] RidgeClassifierCV

(some might have been fixed since, need to check).
 
The required sample weight invariance properties (including the behavior of sw=0) were also discussed in https://github.com/scikit-learn/scikit-learn/issues/15657
user: rth
issue_16270: state: open
title: In PassiveAggressiveClassifier, inconsistency between the parameter "loss" and the attribute "loss_function_"
body: #### Describe the issue linked to the documentation
The two possible loss values for the PassiveAggressiveClassifier are "hinge" and "squared_hinge", but in both cases, the classifier attribute "loss_function_" is the hinge loss, and only the learning rate is adapted. 
``` python
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets

from sklearn.linear_model import PassiveAggressiveClassifier

heldout = [0.95, 0.90, 0.75, 0.50, 0.01]
rounds = 20
X, y = datasets.load_digits(return_X_y=True)

classifiers = [
    ("Passive-Aggressive I", PassiveAggressiveClassifier(loss='hinge',
                                                         C=1.0, tol=1e-4)),
    ("Passive-Aggressive II", PassiveAggressiveClassifier(loss='squared_hinge',
                                                          C=1.0, tol=1e-4))
]

xx = 1. - np.array(heldout)

for name, clf in classifiers:
    print("training %s" % name)
    rng = np.random.RandomState(42)
    yy = []
    for i in heldout:
        yy_ = []
        for r in range(rounds):
            X_train, X_test, y_train, y_test = \
                train_test_split(X, y, test_size=i, random_state=rng)
            clf.fit(X_train, y_train)
            y_pred = clf.predict(X_test)
            yy_.append(1 - np.mean(y_pred == y_test))
       print(clf.loss_function_)
```
returns
```
training Passive-Aggressive I
<sklearn.linear_model._sgd_fast.Hinge object at 0x1172efb30>
training Passive-Aggressive II
<sklearn.linear_model._sgd_fast.Hinge object at 0x1172efcd0>
```



#### Suggest a potential alternative/fix
shouldn't the signature of the classifier reflect the implemented algorithm, and allow the user to set the learning_rate parameter instead of the loss?
Indeed, the user set loss parameter is not used, and loss="hinge" is hardcoded instead of loss=self.loss ([here](https://github.com/scikit-learn/scikit-learn/blob/40f668c03891caaeda231279fe12cbb95b198cbe/sklearn/linear_model/_passive_aggressive.py#L251) and [here](https://github.com/scikit-learn/scikit-learn/blob/40f668c03891caaeda231279fe12cbb95b198cbe/sklearn/linear_model/_passive_aggressive.py#L223))
user: judithabk6
issue_16248: state: open
title: Add crossentropy (or weighted classification loss) for SGDClassifier
body: At present [log loss](https://scikit-learn.org/stable/modules/model_evaluation.html#log-loss) is available as an evaluation metric for classiferiers, but I wonder if there is also interest in including it as an objective function for SGDClassifier.

My motivation stems from weakly supervised classifier training (a la [Snorkel](https://www.snorkel.org/use-cases/01-spam-tutorial#5-training-a-classifier)), where training data contain only probabilistic  labels.

I believe that this can be implemented without changing the signatures of any of the existing methods. In particular, for a binary case, a `WeightedLog` loss class might exploit `Log` loss class. In this case, assume that the argument `y` to the `loss()` method actually carries the probability of the record belonging to the positive class. This explains the return value of the method shown below.
```python
cdef class WeightedLog(Classification):
    """Weighted logistic regression loss for binary classification with y in [0.0, 1.0]"""

    cdef Log log

    def __init__(self):
        self.log = Log()

    cdef double loss(self, double p, double y) nogil:
        """Target label 'y' denotes the probability of positive"""
        if y < 0.0:
            raise ValueError(("Target label 'y' cannot be smaller than"
                              " 0.0 when using 'weighted_log' loss"))
        if y > 1.0:
            raise ValueError(("Target label 'y' cannot be greater than"
                              " 1.0 when using 'weighted_log' loss"))
        return (y * self.log.loss(p, 1)) + ((1 - y) * self.log.loss(p, -1))

    cdef double _dloss(self, double p, double y) nogil:
        if y < 0.0:
            raise ValueError(("Target label 'y' cannot be smaller than"
                              " 0.0 when using 'weighted_log' loss"))
        if y > 1.0:
            raise ValueError(("Target label 'y' cannot be greater than"
                              " 1.0 when using 'weighted_log' loss"))
        return (y * self.log._dloss(p, 1)) + ((1 - y) * self.log._dloss(p, -1))

    def __reduce__(self):
        return WeightedLog, ()
```
In order to enable this, we might have to also modify the method [`_prepare_fit_binary()`](https://github.com/chaturv3di/scikit-learn/blob/c45721d538d36ab4c322d18e33d9c10b55f5fe27/sklearn/linear_model/_stochastic_gradient.py#L290) slightly so that it sets `y_i = y` if `weighted_log` loss is chosen.

At least one down side is that handling `class_weight` becomes hard, if not impossible, while fitting the classifier. I'll be happy to receive feedback.

PS: I can raise a PR if this addition makes sense.
user: chaturv3di
issue_16246: state: open
title: LDA and QDA covariance_ attributes are inconsistent and misleading
body: Both `LDA` and `QDA` have a `covariance_` attribute. There are however a few important differences:

- The attributes don't have the same shape. `lda.covariance_` is of shape (n_features, n_features) while for qda this is a list of arrays of shape (n_features, n_features). The list has K entries where K is the number of classes.
- `lda.covariance_` is **not** a covariance but a weighted *within-class covariance matrix*. It is equal to
`sum_k pi_k * S_k` where `pi_k` is the proportion of samples in class k and `S_k` is the covariance matrix for these samples. The entries of the list in qda correspond to those `S_k`.
- the covariances computed in `lda` are biased, while those computed in `qda` are unbiased (divide by n - 1 instead of by n) (previously noted in https://github.com/scikit-learn/scikit-learn/issues/15640)


Given those discrepancies and the misleading names, I would propose the following:

- deprecate `lda.covariance_` in favor of `lda.within_class_cov_`
- deprecate `qda.covariance_` in favor of `qda.covariances_`.

The docstring would be updated to indicate that the estimators are biased / unbiased.

WDYT?

Side note, I found this while working on #16243 
user: NicolasHug
issue_16241: state: open
title: check_estimator is stricter than what is stated in the Estimator API doc
body: This issue was raised in a discussion regarding the LightGBM scikit-learn compatible estimators: https://github.com/microsoft/LightGBM/issues/2628#issuecomment-574369813

The problem is that check_estimator complains about private attributes set in the `__init__` of a scikit-learn estimator while our documentation just state the following (while not explicitly prohibiting setting private attributes in `__init__`):

> The arguments accepted by `__init__` should all be keyword arguments with a default value. In other words, a user should be able to instantiate an estimator without passing any arguments to it. The arguments should all correspond to hyperparameters describing the model or the optimisation problem the estimator tries to solve. These initial arguments (or parameters) are always remembered by the estimator. Also note that they should not be documented under the “Attributes” section, but rather under the “Parameters” section for that estimator.
>
> In addition, every keyword argument accepted by `__init__` should correspond to an attribute on the instance. Scikit-learn relies on this to find the relevant attributes to set on an estimator when doing model selection.

from: https://scikit-learn.org/0.22/developers/develop.html#instantiation

The strict check is defined in `sklearn.utils.estimator_checks.check_no_attributes_set_in_init`.


user: ogrisel
issue_16240: state: open
title: Add MSLE(Mean Squared Logarithmic Error) to TreeRegressors
body: <!--
If you want to propose a new algorithm, please refer first to the scikit-learn
inclusion criterion:
https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms
-->

#### Describe the workflow you want to enable
```python
from sklearn.tree import DecisionTreeRegressor
model = DecisionTreeRegressor(criterion = 'msle')
```

There are some differences between MSE and MSLE. First, MSLE only cares about the percentual differences. And second, MSLE penalizes underestimates more than overestimates. 

Especially, in business, I frequently meet the situation which should handle underestimation. When we forecast delivery time, for example, data analysts want to calculate delivery time conservatively to avoid VOC. Sometimes, they just added +10 minutes to their model's result. I remind them that they could convert their label y to log scale and put them in the model to compute error as logarithmic error, but they didn't just because it seems difficult. 

I think, if there is more criterion option, logarithmic error, in the basic model like trees, analysts could handle this problem easily.

#### Describe your proposed solution
Add logarithmic error to sklearn/tree/_criterion.c
Add logarithmic error to sklearn/tree/_classes.py
```python
CRITERIA_REG = {"mse": _criterion.MSE, "friedman_mse": _criterion.FriedmanMSE, "mae": _criterion.MAE, "msle": _criterion.MSLE}
```
Add documentation about MSLE or comparison between MSE and MSLE

#### Describe alternatives you've considered, if relevant
Just add documentation about MSLE and MSE, and share how to compute logarithmic error with MSE criterion in trees.

#### Additional context

user: ysunmi0427
issue_16238: state: open
title: TSNE implementation finds variation where there is none
body: #### Describe the bug
Manifold.TSNE gives back a lower-dimensional representation of the input data, which contains variation, even if the input data only contains zeros (it seems to do this in all cases where all input samples are the same).

#### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.manifold import TSNE

f = TSNE(n_components=2)

zero_array = np.zeros((993, 20))
tsne_array = f.fit_transform(zero_array)

```

#### Expected Results
A warning, or all data mapped to the same point.

#### Actual Results

A lower-dimensional array (here (993, 2) shape) with different vectors!
```python
[[-0.21821421 -0.16358966]
 [-0.21803701 -0.16340736]
 [-0.21847947 -0.16356741]
 ...
 [-0.07215934  0.15475279]
 [-0.4158346   0.2995374 ]
 [-0.21808246 -0.16424587]]
```

#### Versions

Python 3.7.5
NumPy 1.17.4
SciPy 1.3.2
Scikit-Learn 0.21.3

<!-- Thanks for contributing! -->

user: JellePiepenbrock
issue_16236: state: open
title: FEA Group aware Time-based cross validation
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
Fixes #14257 
1 . Assumes that `groups` are contiguous . 
2. Split the `groups` into train and test indices  using TimeSeriesSplit 
3. Use these group indices to get the indices for original data.
4. Use the max_train_size parameter to trim the train_array  for each iteration of split.
#### Any other comments?

Question: 
1. Should we have separate parameter  for max_train_size in group vs. actual data ?
2. Should we have separate parameter  for gap in group vs. actual data ?



<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: getgaurav2
issue_16196: state: open
title: Use np.empty_like in PolynomialFeatures for NEP18 support
body: #### Reference Issues/PRs

Followup of my quest to make vaex work with sklearn via NEP18 #14963 / https://github.com/vaexio/vaex/pull/415


#### What does this implement/fix? Explain your changes.

PolynomialFeatures uses `np.empty`, if it uses np.empty_like, I can intercept it via NEP18, so that vaex can use this transformer.

#### Any other comments?

I could push more of these changes on this branch, but I first want to see if this is what people like to see or not.
user: maartenbreddels
issue_16194: state: open
title: Meanshift alternate implementation issue16171
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs

Implements changes pursuant to Issue #16171

#### What does this implement/fix? Explain your changes.

MeanShift predictions are made via MeanShift.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: csbrown
issue_16188: state: open
title: Consider adding a version of multivariate mutual information from the gist below?
body: If I'm not mistaken, there is no version of mutual information for multiple continuous random variables in sklearn.

Quite a few people have been landing on this gist: https://gist.github.com/GaelVaroquaux/ead9898bd3c973c40429

Any thoughts on including some version of this MI calculation here? 

#### Describe the workflow you want to enable

#### Describe your proposed solution

#### Describe alternatives you've considered, if relevant

#### Additional context

user: cottrell
issue_16184: state: open
title: n_jobs greater than 1 causes segmentation fault
body: #### Describe the bug
Setting n_jobs for `sklearn.manifold.TSNE` to anything greater than 1 (including -1) causes a segmentation fault

#### Steps/Code to Reproduce
<!--
Please add a minimal example that we can reproduce the error by running the
code. Be as succinct as possible, do not depend on external data. In short, we
are going to copy-paste your code and we expect to get the same
result as you. -->

Example:
```python
import numpy as np
from sklearn.manifold import TSNE
TSNE(n_components=2,n_jobs=1).fit_transform(np.zeros((100,512)) # Runs fine
TSNE(n_components=2,n_jobs=2).fit_transform(np.zeros((100,512)) # Segmentation fault
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
Would expect it to work with any amount of `n_jobs`

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
Segmentation fault

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.22.1:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
import imblearn; print("Imbalanced-Learn", imblearn.__version__)
-->
System:
    python: 3.7.3 (default, Mar 26 2019, 01:59:45)  [GCC 5.4.0 20160609]
executable: /home/<ME>/.virtualenvs/reid/bin/python3.7
   machine: Linux-4.15.0-39-generic-x86_64-with-Ubuntu-16.04-xenial
```
Python dependencies:
       pip: 20.0.1
setuptools: 41.0.1
   sklearn: 0.22.1
     numpy: 1.16.3
     scipy: 1.2.1
    Cython: 0.29.12
    pandas: 0.25.1
matplotlib: 3.0.3
    joblib: 0.13.2

Built with OpenMP: True
```
N.B. Could very well be related to #12922 but I'm not sure.

<!-- Thanks for contributing! -->

user: JakeCowton
issue_16171: state: open
title: MeanShift predict method is incorrect
body: #### Describe the workflow you want to enable

The `predict` method on the `MeanShift` class takes in a point and returns the nearest "cluster center" from the final iterations of the training data.  However, `MeanShift` is a lazy learner.  The normal process for `fit`ting is to simply store the training data.  The `predict` method should then perform all of the logic that is currently in the `fit` method.

Note that the current implementation is actually a weighted KMeans.  It is not MeanShift.

#### Describe your proposed solution

Fix MeanShift.  I'm happy to do it if there is agreement that this is broken.

user: csbrown
issue_16161: state: open
title: Attach and use sample properties in pipeline with custom transformers
body: #### Describe the issue linked to the documentation

<!--
Tell us about the confusion introduced in the documentation.
-->

Could someone please clarify the difference between [sample properties](https://scikit-learn.org/stable/glossary.html#term-sample-property) and feature properties (I couldn't find this in the documentation) and how they are intended to be used with the current API?

I would like to create a pipeline with custom transformers/estimators inheriting from `BaseEstimator` and `TransformerMixin`. Transformers at the beginning of the pipeline should add features (or sample properties?) to the current sample being transformed. This attached information should then be usable by Transformers/Estimators later in the pipeline. 

Is there already a way to achieve this with the current API? I read about `fit_params` and pandas DataFrame in some Github-issues, but couldn't find an example how this data can be edited and used by transformers, sort of like a shared data that corresponds to the input samples in `X`. However, I guess my understanding of the `**fit_params` isn't correct? Is it even intended to be modified by individual transformers or just passed to all transformers in a pipeline when calling fit and kept read only? 

I am working with heterogeneous data (not [rectangular](https://scikit-learn.org/stable/glossary.html#term-rectangular)) and used python lists and pandas DataFrames as input `X` for my custom transformers which then return modified data `Xt`, which is again a python list or DataFrame. Both options seem to work so far with my custom modules but this probably doesn't adhere to the Estimator API sklearn provides.

I also though of implementing [`get_feature_names`](https://scikit-learn.org/stable/glossary.html#term-get-feature-names) but I don't know how to access the values from those feature names.

Please let me know if there is a standard way to share data between estimators. 
user: fjp
issue_16157: state: open
title: Comparison of ML Algorithms 
body: #### Describe the issue linked to the documentation
It is difficult to select a ML algorithm for given application. Please provide a comparison of ML algorithms showing advantages and limitation of each algorithm
<!--
Tell us about the confusion introduced in the documentation.
-->

#### Suggest a potential alternative/fix
Generate a new section in the documentation to discuss about advantages and limitations of each algorithm.
<!--
Tell us how we could improve the documentation in this regard.
-->

user: Kapilam
issue_16153: state: open
title: Add threshold comparison above arrow in plot_tree
body: I realised after #13971 and also a lecture that I gave that we don't add "True/False", "Yes/No" above the arrow in `plot_tree`. Therefore, we know the criterion for the decision and the further splits but we don't have any idea what path a sample will take (or at least it makes it more difficult to infer it).

I think that we should annotate the arrow depending if the comparison in the node is True/False.

@amueller @NicolasHug @thomasjpfan WDYT?
user: glemaitre
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_16148: state: open
title: TfidfVectorizer handles multiple text columns
body: It's really nice that transformers such as `sklearn.preprocessing.OneHotEncoder` and `sklearn.preprocessing.StandardScaler` can operate on multiple data columns simultaneously.

`sklearn.feature_extraction.text.TfidfVectorizer` on the other hand, can only process one column at a time, so you need to make a new transformer for each text column in your dataset.  This can get a little tedious and in particular makes pipelines more verbose.

It'd be nice if `TfidfVectorizer` could also operate on multiple text columns, using the same settings for each column, perhaps with an option to make one vocabulary per column, or use a shared vocabulary across all the columns.

It might be easiest to implement this as a new class that wraps `TfidfVectorizer` [sagemaker-scikit-learn-extension](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/feature_extraction/text.py#L22) takes this approach.

If this seems like a good idea, I'd be happy to make a PR.
user: zachmayer
issue_16143: state: open
title: PERF predict_proba is slow when n_jobs > 1 for random forests
body: The use of `n_jobs > 1` for small batch can slow down the prediction for forest models. This is probably due to the overhead incurred by using `joblib` (create thread, check system info, ...) which is dominate the runtime compared to the computations. This was reported originally in joblib/joblib#982.

A couple of ideas to solve this:
- Set `n_jobs=1` when the size of the batch is small.
- Introduce a `n_jobs_predict` parameters that would default to `1/n_jobs` for forests but that can be set separately.


user: tomMoral
issue_16135: state: open
title: Using `cross_val_predict` with `RepeatedKFold`
body: It would be nice to use `RepeatedKFold` with with `cross_val_predict`. It's possible use `cross_val_score` with repeated K folds, but a `ValueError` is thrown with `predict`

```
model = LogisticRegression()
rkf = RepeatedKFold(n_splits=10, n_repeats=10)
prediction = cross_val_predict(model, X, y, cv=rkf)
score = cross_val_score(model, X, y, cv=rkf)
```

Throws `ValueError: cross_val_predict only works for partitions`.
user: jacobcvt12
issue_16130: state: open
title: API: do we allow a list of generic objects as X? 
body: https://github.com/scikit-learn/scikit-learn/pull/15557 introduced a change in the GP module to allow a list of generic objects and kernels on those data, which is nice, but it violates our API AFAIK.

If we're going to allow that, there are many other places we can/should allow that as well (SVMs for instance).

But that doesn't really play well with the rest of the library (pipeline, GS, etc).

ping @scikit-learn/core-devs 
user: adrinjalali
issue_16123: state: open
title: Adding implementation of Hartigan's K-Means
body: Hi,

I'm implementing Hartigan's K-Means in C++ with a Cython wrapper, and when it's done I'd be glad to contribute it to scikit-learn. The implementation follows the pseudo code described in the IJCAI '13 paper by Slonim, Aharoni and Crammer (https://dl.acm.org/doi/10.5555/2540128.2540369) + some optimizations of my own that make the run-time comparable to Lloyd's K-Means.

I'd like to know if the community welcomes this addition.

Thank you.
user: assaftibm
issue_16081: state: open
title: Hidden additional iteration in k-means, max_iter appears to be off by two
body: The runtime of k-means (including "full") is not proportional to the maximum number of iterations.
But it clearly should be.

Benchmark code (use the MNIST data set):
```python
from sklearn.cluster import KMeans
import time
seeds=data[np.random.RandomState(0).choice(data.shape[0], 10)]
runtimes = []
for i in range(1, 61):
  kml = KMeans(n_clusters=10, init=seeds, max_iter=i, n_init=1, random_state=0, tol=1e-15, algorithm="full")
  start = time.time()
  kml.fit(data)
  runtime = time.time() - start
  runtimes.append([kml.n_iter_,runtime])
  print("%d\t%d\t%f" % (i, kml.n_iter_, runtime))

import matplotlib.pyplot as plt
runtimes = np.array(runtimes)
plt.plot(runtimes[:,0], runtimes[:,1]/(runtimes[:,0]), label="current")
plt.plot(runtimes[:,0], runtimes[:,1]/(runtimes[:,0]+1), label="+1")
plt.plot(runtimes[:,0], runtimes[:,1]/(runtimes[:,0]+2), label="+2")
plt.legend()
plt.show();
```
Resulting in the following plot.
![kmeans](https://user-images.githubusercontent.com/3997899/72097110-efe33580-331b-11ea-90f2-9f9df2393486.png)

It will converge at 63 iterations; when it converged the iteration count will be okay. This plot indicates that the current k-means algorithm performs TWO more iterations than advertised. And it is not just startup cost, unfortunately.

There is one hidden additional iteration performed (not completely, but most of an iteration) here:
https://github.com/scikit-learn/scikit-learn/blob/50d3fe963c3d2823b2ba2a534b8a50dba620fc5e/sklearn/cluster/_kmeans.py#L452-L455
This is almost the entire cost of an iteration hidden here... (c.f. line 420-423). Hence, the "max_iter" count in these cases is at least "off by one". But it seems there is at least another hidden iteration somewhere else.
user: kno10
issue_16080: state: open
title: Missing argument `ensure_2d=False` in call to check_array()
body: https://github.com/scikit-learn/scikit-learn/blob/4fe4d27eb1b88cc799199f2176ac734bbc16ca31/sklearn/metrics/pairwise.py#L268

According to the docstrings of euclidean_distances(), `X_norm_squared` is 1D.
user: chmendoza
labels: Needs Decision: Requires decision
issue_16062: state: open
title: Add subsample and max_features parameters to HistGradientBoostingRegressor
body: The parameters **subsample** and **max_features** in GradientBoostingRegressor are useful. Is it possible to add equivalent parameters to HistGradientBoostingRegressor?
user: Gitman-code
issue_16055: state: open
title: Add multiclass support for brier_score_loss
body: Feature Request: The original formulation for Brier score inherently supports multiclass classification [[source](https://en.wikipedia.org/wiki/Brier_score#Original_definition_by_Brier)]. This is currently absent in scikit-learn, which restricts Brier score to binary classification. 

I have found it useful to have multiclass brier_score_loss when comparing calibration, and have a custom function for the same.  I would be happy to work on this and port the code to the scikit-learn API.
user: aggvarun01
issue_16051: state: open
title: [ENH] ICA based on AJD (SOBI, AMUSE, NSS)
body: While developing an extension of ICA to grouped data (as opposed to simply pooling data), we (@NiklasPfister & myself) also implemented a variety of related linear ICA algorithms.
They are based on approximate joint matrix diagonalization and, as opposed to FastICA, do not rely on non-Gaussianity of the sources. In particular, the following ICA-algorithms are implemented for the right choice of parameters by our UwedgeICA sklearn transformer:
* NSS-JS, NSS-TD-JD (NSS = non-stationarity in time or space) as well as
* the popular SOBI (second-order) and its predecessor AMUSE

(references can be found in the paper below).
Additionally, the CoroICA transformer implements confounding-robust extensions of those aforementioned methods.

* [Code](https://github.com/sweichwald/coroICA-python/)
* [JMLR Paper](http://www.jmlr.org/papers/v20/18-399.html)
* [Website](https://sweichwald.de/coroICA/)
* [Documentation](https://sweichwald.de/coroICA-python/)
* We are attaching a MWE below.

We guess it should be quite doable to include those in sklearn as our current implementation is already sklearn flavoured and built upon the respective mixins, PEP8 compatible, and comes with examples and documentation. Also the sklearn examples available for FastICA could be easily adapted and extended to demo the different ICA variants.

We are raising the issue to get early feedback and gauge the interest — what do you think?

@GaelVaroquaux @agramfort

--

Get current implementation: `pip install coroica`.

```python
from coroica import UwedgeICA
from coroica.utils import md_index
import numpy as np
from sklearn.decomposition import FastICA


np.random.seed(0)


# generate 2 sources with changing variance
S = np.random.randn(1000, 2) \
    * np.c_[np.sin(np.arange(1000) / 1000),
            np.cos(np.arange(1000) / 1000)]
A = np.random.randn(2, 2)
X = (A.dot(S.T)).T

# transformers
fastica = FastICA(random_state=0)
uwedgeica = UwedgeICA(partitionsize=50)

# fit both ICAs
fastica.fit(X)
uwedgeica.fit(X)

print(f'MD(A,V_fastica) = {md_index(A, fastica.components_):.2f}')
print(f'MD(A,V_uwedge) = {md_index(A, uwedgeica.V_):.2f}')

# yields:
# MD(A,V_fastica) = 0.04
# MD(A,V_uwedge) = 0.02
```
user: sweichwald
issue_16050: state: open
title: [Feature Request] Add class_weight parameter for multiclass wrappers
body: #### Description
Most (perhaps all?) binary classifiers in scikit-learn can be generalized to multiclass classification with a `multi_class` parameter. These classifiers also have a `class_weight` parameter to give different weights to different classes.

However, multiclass wrappers such as  [OneVsOneClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.multiclass.OneVsOneClassifier.html#sklearn.multiclass.OneVsOneClassifier) and [OneVsRestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.multiclass.OneVsRestClassifier.html#sklearn.multiclass.OneVsRestClassifier) do not have a `class_weight` parameter, which means that only uniform weights (`None`) and balanced weights (`'balanced'`) provided to the binary classifier are available.

The main use case would be to generalize custom binary classifiers to multiclass classification with a `class_weight` parameter.

#### Steps/Code to Reproduce
Since this works:

```python
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression

X, y = make_classification(n_classes=4, n_clusters_per_class=1)
clf = LogisticRegression(multi_class='ovr', class_weight={0: 1, 1: 3, 2: 5, 3: 6})
clf.fit(X, y)
```

one could expect this to produce the same results:
```python
from sklearn.multiclass import OneVsRestClassifier

clf = OneVsRestClassifier(
    LogisticRegression(), class_weight={0: 1, 1: 3, 2: 5, 4: 6}
)
clf.fit(X, y)
```

#### Proposed changes

Looking briefly at the source code, I would say that it would required to change the `_fit_binary` function by adding a `class_weight` parameter that would be passed to the estimator when it is fitted (the trick would be to provide the right (key, value) dictionary, taking into account the encoding of the labels) :
https://github.com/scikit-learn/scikit-learn/blob/b194674c42d54b26137a456c510c5fdba1ba23e0/sklearn/multiclass.py#L65-L80

#### Versions

```
System:
    python: 3.7.3 (default, Mar 27 2019, 16:54:48)  [Clang 4.0.1 (tags/RELEASE_401/final)]
executable: /Users/johann.faouzi/anaconda3/bin/python
   machine: Darwin-18.7.0-x86_64-i386-64bit

Python dependencies:
       pip: 19.1.1
setuptools: 41.0.1
   sklearn: 0.22.1
     numpy: 1.16.4
     scipy: 1.3.0
    Cython: 0.29.14
    pandas: 0.25.3
matplotlib: 3.1.0
    joblib: 0.13.2

Built with OpenMP: True
```
user: johannfaouzi
issue_16041: state: open
title: Wrongly implemented test in RidgeCV
body: The following test does not actually test anything:

https://github.com/scikit-learn/scikit-learn/blob/bdf1ae9e4a739abf06076e67dc453774ffc886a7/sklearn/linear_model/tests/test_ridge.py#L478-L502

`gcv_ridge` will compute the error by aggregating all predictions and compute the `explained_variance` on all predictions.

However, `loo_ridge` will make a `GridSearchCV` which will compute the score for each individual sample (because of a LOO CV) and then report the mean. However, the explained variance will always be 1.0 since we are computing the variance on a single sample.

So the test is passing by chance.
user: glemaitre
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_16029: state: open
title: [WIP] BUG returned most regularized model with Ridge*CV in case of tie
body: required merging #14848 first 

@agramfort pointed out that Ridge*CV should pick up the most regularized model in case of a tie to be consistent with LassoCV as well.

This PR intends to solve this issue by sorting the `alphas` in descending order before to perform the search.
user: glemaitre
issue_16017: state: open
title: TfidfVectorizer ngrams does not work when vocabulary provided
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description

The `TfidfVectorizer` does not honor the `ngram_range` argument when the `vocabulary` is provided.

#### Steps/Code to Reproduce

Example 1, vocabulary is *not* provided, this works as expected:

```python
from sklearn.feature_extraction.text import TfidfVectorizer

X = ['abc',
     'bcd',
     'cde']

tfidf = TfidfVectorizer(stop_words=None,
                        analyzer='char',
                        ngram_range=(2, 2))

sps = tfidf.fit_transform(X)
print(tfidf.get_feature_names())
# ['ab', 'bc', 'cd', 'de']
```

Example 2, when vocabulary is provided. This does *not* work as expected:

```python
from sklearn.feature_extraction.text import TfidfVectorizer

X = ['abc',
     'bcd',
     'cde']

tfidf = TfidfVectorizer(stop_words=None,
                        analyzer='char',
                        vocabulary={'a': 0, 'b': 1, 'c': 2, 'd': 3, 'e': 4},
                        ngram_range=(2, 2))

sps = tfidf.fit_transform(X)
print(tfidf.get_feature_names())
# ['a', 'b', 'c', 'd', 'e']
```

Note that it works if the vocabulary I provide are the ngrams themselves:

```python
from sklearn.feature_extraction.text import TfidfVectorizer

X = ['abc',
     'bcd',
     'cde']

tfidf = TfidfVectorizer(stop_words=None,
                        analyzer='char',
                        vocabulary=['ab', 'bc', 'cd', 'de'],
                        ngram_range=(2, 2))

sps = tfidf.fit_transform(X)
print(tfidf.get_feature_names())
# ['ab', 'bc', 'cd', 'de']
```

But that seems kind of silly, since I can't possibly know all of the ngrams a priori for a large dataset.

#### Expected Results

Expected to still get ngrams when vocabulary is provided, but did not.

#### Actual Results

See steps to reproduce above.

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->

```
System:
    python: 3.7.5 (default, Oct 25 2019, 10:52:18)  [Clang 4.0.1 (tags/RELEASE_401/final)]
executable: /anaconda3/envs/myenv/bin/python
   machine: Darwin-18.6.0-x86_64-i386-64bit

Python dependencies:
       pip: 19.3.1
setuptools: 42.0.2.post20191203
   sklearn: 0.22
     numpy: 1.17.4
     scipy: 1.4.0
    Cython: 0.29.14
    pandas: 0.25.3
matplotlib: 3.1.2
    joblib: 0.14.1

Built with OpenMP: True
```


<!-- Thanks for contributing! -->
user: tgsmith61591
issue_16012: state: open
title: [RFC] return Series for target when using as_frame and return_X_y
body: This a general question regarding the API of `as_frame` and `return_X_y` with our loader/fetcher. We have an inconsistent behaviour which could be solved straight away.

`fetch_opeml` introduced `as_frame` exposing `frame` attribute in the `Bunch` which is a Pandas DataFrame. In conjunction with `return_X_y=True`, we exposed `data` and `target`. `data` will always be a `DataFrame` while `target` is supposed to be a `DataFrame` or a `Series` depending on the number of columns in the `target`.

```python
X, y = fetch_openml('iris', as_frame=True, return_X_y=True)
y
```
```
0         Iris-setosa
1         Iris-setosa
2         Iris-setosa
3         Iris-setosa
4         Iris-setosa
            ...      
145    Iris-virginica
146    Iris-virginica
147    Iris-virginica
148    Iris-virginica
149    Iris-virginica
Name: class, Length: 150, dtype: category
Categories (3, object): [Iris-setosa, Iris-versicolor, Iris-virginica]
```

In https://github.com/scikit-learn/scikit-learn/pull/15950, we introduce `as_frame` to `fetch_california_housing`. The API is the same apart of the `target` output. The `target` is a `DataFrame` even with a single column.

```python
X, y = fetch_california_housing(return_X_y=True, as_frame=True)
y
```
```
       MedHouseVal
0            4.526
1            3.585
2            3.521
3            3.413
4            3.422
...            ...
20635        0.781
20636        0.771
20637        0.923
20638        0.847
20639        0.894

[20640 rows x 1 columns]
```

So my question is: what type of `target` do we want when the target is 1D?
user: glemaitre
labels: Needs Decision: Requires decision
issue_16010: state: open
title: [WIP] Kernel cookbook -- additional kernels for Gaussian Process Regression
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

#### What does this implement/fix? Explain your changes.
Added support for all kernels in the "kernel cookbook" [https://www.cs.toronto.edu/~duvenaud/cookbook/](https://www.cs.toronto.edu/~duvenaud/cookbook/) 
except the low-rank projection (could be added, superseded by variational methods such as implemented in GPy, which would require additional likelihood functions).

Especially note the use of `ProjectionKernel` with sets of indicator variables, (i.e. one-hot-encoded) for use with categorical variables.  `ProjectionKernel` is comparable in utility to GPy's `active_dims`.  Typically one constructs X from continuous variables concatenated with one or more sets of dummy-coded columns.  constructs an RBF kernel over the projection onto each dummy-coded set, constructs any desired kernel over the projection onto the continuous variables, and these may then be combined using Sum, Product, etc.  

For example if X has 3 continuous variables concatenated with three columns output by one-hot encoding, using a RationalQuadratic kernel over the continuous variables and defaulting to RBF over the encoded categorical variable:
`k = ProjectionKernel(columns=[0,1,2], name='continuous', kernel=RationalQuadratic()) * ProjectionKernel(columns=[3,4,5], name='categorical')`
(The `name` parameter is used to label the parameters.)

`CompoundKernel` is extended to include `DirectSum` and `Tensor` (DirectProduct) using `reduce`.
- `CompoundKernel` was not fully implemented; the missing methods have now been added.
- fixed a bug in `set_param` referencing undefined `.k1`
- added `.n_dims`
- added support for `deep` in `get_param`

#### Any other comments?
- No unit tests at present.
- Comments 
-- are not yet complete,
-- do not yet follow standards everywhere,
-- some are out of date.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: JohnHBauer
issue_15992: state: open
title: Integration testing for downstream dependencies
body: We need a new Continuous Integration configuration to launch the tests of some popular downstream projects.

Candidate projects:

- xgboost and or lightgbm as very popular third party libraries
- skorch
- imbalanced-learn not necessarily as popular as others but with high use of sklearn's API
- autosklearn?
- any other suggestion?

Those test would not be run in PRs unless we use a specific tag in the commit message for instance. This test setup would be run by maintainers in the release branches.

We do something similar in the cloudpickle project with the `[ci downstream]` commit tag:

https://github.com/cloudpipe/cloudpickle/blob/master/.travis.yml#L37-L70

For scikit-learn this would probably need a bit more scripting as the build is more complex than for a pure python project such as cloudpickle but the general idea would be the same.
user: ogrisel
issue_15985: state: open
title: Filter empty samples after removing metadata from 20 newsgroup dataset
body: I was trying out some models on the 20 newsgroup dataset by using the `fetch_20newsgroups` method, but was geting some blank samples, which was causing my preprocessing and models to fail. Then I figured out that the `remove` option may remove all the content of some samples, as can be seen bellow. So I'm proposing to filter out those empty samples after removing the metadata to avoid silently generate blank samples.

```python
from sklearn.datasets import fetch_20newsgroups

full = fetch_20newsgroups(subset='train')
removed = fetch_20newsgroups(subset='train', remove=('headers', 'footers', 'quotes'))

print(len(full.data[155])) # => 849
print(len(removed.data[155])) # => 0
```
user: hugoabonizio
issue_15982: state: open
title: Affinity propagation preference percentile
body: Reference Issues/PRs
Note: Since my last pull request  for this feature was too old, I have redone it.
I would love if someone can guide me so I can finish this feature.

What does this implement/fix? Explain your changes.
This implements a preference_percentile parameter for the affinity propagation clustering module.
It gives control over the number of classes regardless of the size of the similarities.

Any other comments?
The authors of Affinity propagation suggest using a function to find the min and max of the similarities in order to compute the range of the preference parameter and thus controlling the number of clusters. implementing the preference_percentile allowing to change the preference variable within the range, without knowing computing the similarities matrix outside of the affinity propagation module.


user: GuyPozner
issue_15953: state: open
title: MultiOutputRegressor: Support for more fit parameters
body: #### Description

This is a feature wanted. Till latest version of `sklearn`, the `MultiOutputRegressor.fit ` only support a optional `sample_weight` parameter.   It would be nice if it support another optional `fit_param` parameter, which will enhance the `estimator.fit`. For example, we can use `lightgbm` or `xgboost` early stopping fitting way to overcome the over-fitting  issue. 

I know it is a little bit complicated to realize that. But I I hope you will consider that. Thanks!

#### Steps/Code to Reproduce

This is my expected usage example.

```python
#!/usr/bin/env python3

import numpy as np
from sklearn.multioutput import MultiOutputRegressor
import lightgbm as lgb

train_X = np.random.random((10, 10))
train_y = np.random.random((10, 4))
eval_X = np.random.random((10, 10))
train_y = np.random.random((10, 4))
single_model = lgb.GBMRegressor()
model = MultiOutputRegressor(single_model)
fit_param = {'verbose': False, 'early_stopping_rounds':10, 'eval_set':(eval_X, eval_y)}
reg.fit(train_X, train_y, fit_param=fit_param)
```
#### Expected Results

Unsupported yet.

#### Actual Results

Unsupported yet.

#### Versions

```python
Scikit-Learn: 0.22
pltaform: Windows-10-10.0.14393-SP0
python: 3.6.9
```
user: huangk10
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_15932: state: open
title: compute pairwise_distance with custom metric function for non-numeric data
body: This is a feature request: I'm working with strings and have a custom metric function to compute similarities between the strings. I would love to use the pairwise_distance function together with this custom metric to compute the whole distance/similarity matrix for my data. However, I'm getting a `ValueError: could not convert string to float: 'some string'` when the X and Y arrays are checked. It would be great if these checks could be made optional for custom metric functions.
user: cod3licious
issue_15931: state: open
title: [WIP] cd_fast speedup
body: Rewriting implementation to remove redundant calculations.
Old implementation:
1. R = R + w_ii * X[:,ii]
2. tmp = X[:,ii] dot R
3. f(tmp,...)
4. R = R - w[ii] * X[:,ii]

New implementation:
substitute R from 1 into 2 ->
tmp = X[:,ii] dot (R + w_ii * X[:,ii]) 
tmp = X[:,ii] dot R + w_ii * X[:,ii] dot X[:,ii]
2. tmp = X[:,ii] dot R + w_ii * norm_cols_X[ii]
Then to update R:
4. R = R + (w_ii - w[ii]) * X[:,ii]

This removes step one, and rewrites step 2 and 4, improving loop speed.
The method here is probably also extendable to the other 3 functions.

Sadly my python skills are not good enough to build and test this, I did however test this in C++, so everything should work.
Help with building, testing, and extending is very much appreciated.

Tasklist:
- [ ] Build and validate results.
- [ ] Extend to sparse, gram, and multi_task
user: Tjalling-a-a
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_15922: state: open
title: [WIP] Add `sparse-rbf` kernel option for `semi_supervised.LabelSpreading`
body: #### Reference Issues/PRs
This pull request adds another named kernel option, `sparse-rbf` for `semi_supervised.LabelSpreading`.

Loosely related to #15868.

#### Rationale
This kernel provides RBF weights for only the k-nearest-neighbors of each point.

Currently, the only kernel options are `knn` and `rbf` - but for a large dataset, the dense `rbf` kernel option is not feasible (for `N` items, we must compute a dense `NxN` matrix). For any non-trivial dataset, computing the dense kernel matrix will be infeasible, so my purpose here is to give a kernel that can perform better than `knn`, and still be feasible for a large dataset. 

The intuition is that a weighted adjacency matrix gives more information about the graph structure of our dataset, and therefore with appropriate parameter tuning, such a kernel should perform better than a binary adjacency matrix. Furthermore, filling with the RBF weights is cheap, once we have found the k-nearest-neighbors, so the additional runtime cost is minimal.

In particular, I believe that the performance difference will be more clear for datasets with more "difficult" structure; therefore I included CIFAR10 as an option for the example script (the easiest way I know to obtain this is via https://pytorch.org/docs/stable/torchvision/datasets.html#cifar), though I have not yet had time to experiment with a significant fraction of that dataset.

It might also be useful to try this on some toy datasets like https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html, but I have not experimented with this yet.

#### Notes and Questions
I included tests to show:
1. this kernel "works" (the learned classifier has high accuracy)
2. the weights found using `sparse-rbf` are indeed the same as the top-k entries of the kernel that you get using the dense `rbf` option

Since I am making a claim/intuition about performance, I also included an example that does a hyperparameter grid search for the two available sparse kernels (`knn`, already provided, and the new `sparse-rbf`), and then uses the optimal parameters across a range of % supervision to compare:

1. transductive accuracy (guessing labels for the unlabeled training data)
2. inductive accuracy (labeling test data)
3. runtime

I'm opening this PR to get some feedback on the following items:

1. Does this kernel seem helpful, and could I make it more helpful somehow?
2. Is the example I provided clear enough, and would other examples be useful to show the potential benefit of this kernel?
3. For sklearn examples in general, I'm not sure (practically speaking) how to get the results to be visible as embedded plots, so is there something I should do to structure my example appropriately?
4. Are there other experiments/examples that would be useful for evaluating this kernel that folks would like to see?

#### Caveats about the hyperparameter search
- The search is done using only the **inductive accuracy** because of the API used by `GridSearchCV`. It might be possible (with some headache) to run the grid search using transductive accuracy, but this also might not matter.
- The search is done at a fixed % supervision, again because of the API for `GridSearchCV`.
- Notice that the exact hyperparameters found depend on what dataset and what fraction of that dataset is used for the grid search.

#### Example results

Here are the results of using 20% of MNIST, with the fixed parameters included at the bottom of the example script.:
![sparse_kernel_comparison](https://user-images.githubusercontent.com/29764926/71132447-79905f00-21c5-11ea-9e7d-f3dc9b0abde7.png)


Please let me know if you need more info.
Thanks!

user: nik-sm
issue_15919: state: open
title: str categorical data  :  CategoricalNB needed classes_array to understand in which classes order are arrays for feature_log_prob_ 
body: 1 
great code for CategoricalNB thanks
we waited for long time
2
only
for CategoricalNB needed classes_array to understand in which classes order are arrays for feature_log_prob_  for target defined as str
for example : good and bad
or
green and red

may be needed Attribute classes_array
like for  LogisticRegression
https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
Attributes

classes_array, shape (n_classes, )

A list of class labels known to the classifier.
3
also can you add capability to send str format categorical data as it is
for now we use LabelEncoder to transform str categorical data to integers

thanks a lot in advance ...
 


user: Sandy4321
issue_15917: state: open
title: Consider using extension-helpers package for OpenMP helpers
body: @Cadair and I were pleased to notice that you use a fork of the openmp_helpers.py file from astropy in scikit-learn:

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/_build_utils/openmp_helpers.py

We just wanted to let you know that we have been working on generalizing the helpers we use in astropy and have now released a lightweight and general package called [extension-helpers](https://extension-helpers.readthedocs.io) which includes the OpenMP helpers. This package is intended to be used as a build-time dependency in a ``pyproject.toml`` file if/once you switch to using this to specify build-time requirements.

Would you be interested in contributing your fixes to openmp_helpers.py upstream to extension-helpers, and using extension-helpers as a build-time dependency? Let us know if you have any questions about extension-helpers! (the GitHub repository is at https://github.com/astropy/extension-helpers).
user: astrofrog
labels: Needs Decision: Requires decision
issue_15910: state: open
title: Crossvalidation on estimator fails when using option multioutput = 'raw_values' in make_scorer function
body: 

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description

I wanted to perform a cross validation of a PLS2 model. Therefore I used the mean_squared_error as scorer function.
As I'm interested in the individual error of individual futures in Y, I used option multioutput = 'raw_values' in the definition of the 'make_scorer' function.
Python throws the error: 
```
  File ""C:\Users\___\Anaconda3\envs\main\lib\site-packages\sklearn\model_selection\_validation.py", line 602, in _score
    raise ValueError(error_msg % (score, type(score), name))
ValueError: scoring must return a number, got [0.00317655 0.00193264 0.00349379 0.00108055] (<class 'numpy.ndarray'>) instead. (scorer=score)
```

#### Steps/Code to Reproduce

```python
# import moduls
import numpy as np
from sklearn.cross_decomposition import PLSRegression 
from sklearn.metrics import make_scorer, mean_squared_error
from sklearn.model_selection import cross_val_score
from scipy.stats import norm

# create X and Y data sets: X is the sum of ks Gaussians with the Amplitude factor in Y
ns = 100 # number of samples
ms = 100   # number of x futures
ks = 4 # number of y futures
x = np.zeros([ns,ms]) # X data set
y = np.zeros([ns,ks]) # Y data set
x_scale = np.linspace(-3.,3.,ms)
x_gauss_center = np.random.uniform(-3.,3.,ks)
x_gauss_scale = 0.1 
for n in range(ns):
    x_gauss_amp = np.random.uniform(0,1.,ks)
    x[n,:] = np.sum([ norm.pdf((x_scale-xgc)/x_gauss_scale)*y_par for xgc,y_par in zip(x_gauss_center,x_gauss_amp)],0)
    y[n,:] = [y_par for y_par in x_gauss_amp]
    ''
# add some noise to the data sets
x_noise = 0.005
y_noise =0.005
x += np.random.normal(0.,x_noise,[ns,ms])
y += np.random.normal(0.,y_noise,[ns,ks])

# prepare the model
regression_model = PLSRegression(n_components = ks)
regression_model.fit(x,y)

# prepare scorers: mean_sqaured_error with option multioutput
mse_scorer_object_passes = make_scorer(
                                mean_squared_error, 
                                greater_is_better = True, 
                                multioutput = 'uniform_average'
                    )
mse_scorer_object_fails = make_scorer(
                                mean_squared_error, 
                                greater_is_better = True, 
                                multioutput = 'raw_values'
                    )

# passes: do crossvalidation with multioutput = 'uniform_average'
mse_per_block =  cross_val_score(regression_model, 
                                x, 
                                y, 
                                cv = 5, 
                                scoring = mse_scorer_object_passes)

# fails do crossvalidation with multioutput = 'raw_value'
mse_per_block =  cross_val_score(regression_model, 
                                x, 
                                y, 
                                cv = 5, 
                                scoring = mse_scorer_object_fails)
```

#### Expected Results
a numpy array with size (  cv = 5 -> <number_of_folds>, ks <y_futures>)

#### Actual Results
An error:
```
  File "C:\Users\___\Anaconda3\envs\main\lib\site-packages\sklearn\model_selection\_validation.py", line 602, in _score
    raise ValueError(error_msg % (score, type(score), name))
ValueError: scoring must return a number, got [0.00317655 0.00193264 0.00349379 0.00108055] (<class 'numpy.ndarray'>) instead. (scorer=score)
```

#### Versions
System:
    python: 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 22:01:29) [MSC v.1900 64 bit (AMD64)]
executable: C:\Users\___\Anaconda3\envs\main\python.exe
   machine: Windows-10-10.0.18362-SP0

Python dependencies:
       pip: 19.3.1
setuptools: 42.0.2.post20191201
   sklearn: 0.22
     numpy: 1.17.3
     scipy: 1.3.1
    Cython: None
    pandas: 0.25.2
matplotlib: 3.1.1
    joblib: 0.14.0

Built with OpenMP: True



user: gstiefel
issue_15900: state: open
title: How to do LeaveOneOut cross validation
body: Currrently, we have two ways to do LOO cv in scikit-learn
The first one is in GridSearchCV, where we calculate the score of each fold (i.e., each sample) and then take the average.
The second one is in RidgeCV, where we calculate the prediction of each fold (i.e., each sample), put them together and calculate the score.
I think this inconsistency is annoying.
Another issue is that whether we should consider sample_weight when averaging the scores in the first option and when calculating the scores in the second option. We do so in RidgeCV, but don't do so in GridSearchCV.
Related to RidgeCV issues ping @glemaitre 
user: qinhanmin2014
issue_15894: state: open
title: Feature Request: Include Heterogeneous Distance Metrics  
body: Would it be possible to include non-numeric and heterogeneous distance metrics as part of the standard library?  

Unfortunately, none of the distance calculations designed to handle categorical features or mixed categorical/numeric datasets appear to be within the current metrics and the overhead of calling a user-defined function instead is tremendous.

There is an excellent package that has already implemented some of these here https://github.com/KacperKubara/distython that I hope would provide a solid starting point.

Thanks very much

user: pynkfloydfan
issue_15881: state: open
title: Bias correction for TransformedTargetRegressor
body: #### Description
If one is interested in predicting the  (conditional on `X`) expected value of a target `y`, aka mean, then `TransformedTargetRegressor` should use a bias corrected inverse transform.

It would be nice to have an option for bias correction in `TransformedTargetRegressor`. At least, I would mention this in the user guide.

#### References
https://robjhyndman.com/hyndsight/backtransforming/

user: lorentzenchr
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_15873: state: open
title: make_column_selector: support for cardinality selection
body: `make_column_selector` is great addtion in 0.22 but one feature is missing, namely selection by cardinality. For example we want to one-hot encode only categorical columns that has low number of unique values and target-encode the rest.  
This feature seems to require only small addition to code-base:

```
class make_column_selector():
    def __init__(self, pattern=None, dtype_include=None, dtype_exclude=None,
                 cardinality=None, cardinality_treshold=7):
        self.pattern = pattern
        self.dtype_include = dtype_include
        self.dtype_exclude = dtype_exclude

    def __call__(self, df):
        if not hasattr(df, 'iloc'):
            raise ValueError("make_column_selector can only be applied to "
                             "pandas dataframes")
        df_row = df.iloc[:1]
        if self.dtype_include is not None or self.dtype_exclude is not None:
            df_row = df_row.select_dtypes(include=self.dtype_include,
                                          exclude=self.dtype_exclude)
        cols = df_row.columns
        if self.pattern is not None:
            cols = cols[cols.str.contains(self.pattern, regex=True)]
        
        cols = list(cols)
        
        if self.cardinality:
            cols_cardinality = dict(df[cols].nunique())
        if self.cardinality == 'high':
            cols = [c for c in cols_cardinality if cols_cardinality[c] > self.cardinality_treshold]
        if self.cardinality == 'low':
            cols = [c for c in cols_cardinality if cols_cardinality[c] <= self.cardinality_treshold]
        
        return cols
```
user: mglowacki100
labels: Easy: Well-defined and straightforward way to resolve
issue_15865: state: open
title: Test __array_function__ not called in non-estimator API
body: See #14702 where this was fixed in common estimator tests, but not in functions such as cross_validation and permutation_importance
user: jnothman
labels: module:test-suite: everything related to our tests
issue_15856: state: open
title: test_svm_equivalent_sample_weight_C should use tighter tolerance
body: This PR tweaks ``sklearn/svm/tests/test_svm.py::test_svm_equivalence_sample_weight_C`` to specify tighter SVC tolerance parameter, specifically `tol=1e-6`, as well as absolute tolerance in `assert_allclose` to `atol=1e-5`.

This change fixes the failure of the test, when SVC class is patched to use DAAL in daal4py project.

```
_____________________ test_svm_equivalence_sample_weight_C _____________________

    def test_svm_equivalence_sample_weight_C():
        # test that rescaling all samples is the same as changing C
        clf = svm.SVC()
        clf.fit(X, Y)
        dual_coef_no_weight = clf.dual_coef_
        clf.set_params(C=100)
        clf.fit(X, Y, sample_weight=np.repeat(0.01, len(X)))
>       assert_allclose(dual_coef_no_weight, clf.dual_coef_)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0
E       
E       Mismatch: 100%
E       Max absolute difference: 0.00413014
E       Max relative difference: 0.01060894
E        x: array([[-0.44591 , -0.393437, -0.445884,  0.391153,  0.447039,  0.447039]])
E        y: array([[-0.448086, -0.389307, -0.447859,  0.387865,  0.448694,  0.448693]])

miniconda/envs/bld/lib/python3.6/site-packages/sklearn/svm/tests/test_svm.py:493: AssertionError
```

user: oleksandr-pavlyk
labels: Needs Decision - Close: Requires decision for closing
issue_15854: state: open
title: FIX provide predictions in the original space in RidgeCV
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
closes #13998


#### What does this implement/fix? Explain your changes.

The predictions returned by `RidgeCV` are not in the original space but in the normalized space (mean removed and divided by the std. dev.). One has to always denormalize them to be able to compute a score. This PR fixes this issue.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: glemaitre
issue_15836: state: open
title: RANSACRegressor should use weighted loss functions
body: The current loss functions disregard weights.

See #14325
user: jnothman
issue_15825: state: open
title: test_unreachable_accuracy can be reached on aarch64
body: https://cloud.drone.io/conda-forge/scikit-learn-feedstock/44/2/2

```
__________________________ test_unreachable_accuracy ___________________________
[gw6] linux -- Python 3.7.3 $PREFIX/bin/python

    def test_unreachable_accuracy():
        assert_array_almost_equal(
            orthogonal_mp(X, y, tol=0),
            orthogonal_mp(X, y, n_nonzero_coefs=n_features))
    
        assert_array_almost_equal(
            assert_warns(RuntimeWarning, orthogonal_mp, X, y, tol=0,
>                        precompute=True),
            orthogonal_mp(X, y, precompute=True,
                          n_nonzero_coefs=n_features))

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.7/site-packages/sklearn/linear_model/tests/test_omp.py:79: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

warning_class = <class 'RuntimeWarning'>
func = <function orthogonal_mp at 0xffff965950d0>
args = (array([[ 2.91575919e+00,  1.14685480e+00,  1.81414114e+00,
         3.47278472e+00,  3.69900930e+00, -1.89437288e+00,...   0.06811033],
       [  2.63341917,   0.14703522,   1.76173811],
       [  2.03115852,  -3.98034558,  -6.03227139]]))
kw = {'precompute': True, 'tol': 0}, w = []
result = array([[ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  ...       ,  0.        ],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ]])

    def assert_warns(warning_class, func, *args, **kw):
        """Test that a certain warning occurs.
    
        Parameters
        ----------
        warning_class : the warning class
            The class to test for, e.g. UserWarning.
    
        func : callable
            Callable object to trigger warnings.
    
        *args : the positional arguments to `func`.
    
        **kw : the keyword arguments to `func`
    
        Returns
        -------
    
        result : the return value of `func`
    
        """
        with warnings.catch_warnings(record=True) as w:
            # Cause all warnings to always be triggered.
            warnings.simplefilter("always")
            # Trigger a warning.
            result = func(*args, **kw)
            if hasattr(np, 'FutureWarning'):
                # Filter out numpy-specific warnings in numpy >= 1.9
                w = [e for e in w
                     if e.category is not np.VisibleDeprecationWarning]
    
            # Verify some things
            if not len(w) > 0:
                raise AssertionError("No warning raised when calling %s"
>                                    % func.__name__)
E               AssertionError: No warning raised when calling orthogonal_mp

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.7/site-packages/sklearn/utils/_testing.py:133: AssertionError
```
user: hmaarrfk
labels: arch:arm: ARM related issues
module:test-suite: everything related to our tests
issue_15821: state: open
title: test_uniform_grid[barnes_hut] can fail on aarch64
body: ```
________________________ test_uniform_grid[barnes_hut] _________________________

method = 'barnes_hut'

    @pytest.mark.parametrize('method', ['barnes_hut', 'exact'])
    def test_uniform_grid(method):
        """Make sure that TSNE can approximately recover a uniform 2D grid
    
        Due to ties in distances between point in X_2d_grid, this test is platform
        dependent for ``method='barnes_hut'`` due to numerical imprecision.
    
        Also, t-SNE is not assured to converge to the right solution because bad
        initialization can lead to convergence to bad local minimum (the
        optimization problem is non-convex). To avoid breaking the test too often,
        we re-run t-SNE from the final point when the convergence is not good
        enough.
        """
        seeds = [0, 1, 2]
        n_iter = 500
        for seed in seeds:
            tsne = TSNE(n_components=2, init='random', random_state=seed,
                        perplexity=20, n_iter=n_iter, method=method)
            Y = tsne.fit_transform(X_2d_grid)
    
            try_name = "{}_{}".format(method, seed)
            try:
>               assert_uniform_grid(Y, try_name)

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.8/site-packages/sklearn/manifold/tests/test_t_sne.py:784: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

Y = array([[ 52.326397  , -15.92225   ],
       [ 46.679527  , -20.175953  ],
       [ 40.870537  , -24.181147  ],
       ...[-35.291374  ,  22.122814  ],
       [-42.2738    ,  18.793724  ],
       [-48.922283  ,  15.606232  ]], dtype=float32)
try_name = 'barnes_hut_1'

    def assert_uniform_grid(Y, try_name=None):
        # Ensure that the resulting embedding leads to approximately
        # uniformly spaced points: the distance to the closest neighbors
        # should be non-zero and approximately constant.
        nn = NearestNeighbors(n_neighbors=1).fit(Y)
        dist_to_nn = nn.kneighbors(return_distance=True)[0].ravel()
        assert dist_to_nn.min() > 0.1
    
        smallest_to_mean = dist_to_nn.min() / np.mean(dist_to_nn)
        largest_to_mean = dist_to_nn.max() / np.mean(dist_to_nn)
    
        assert smallest_to_mean > .5, try_name
>       assert largest_to_mean < 2, try_name
E       AssertionError: barnes_hut_1
E       assert 6.67359409617653 < 2

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.8/site-packages/sklearn/manifold/tests/test_t_sne.py:807: AssertionError

During handling of the above exception, another exception occurred:

method = 'barnes_hut'

    @pytest.mark.parametrize('method', ['barnes_hut', 'exact'])
    def test_uniform_grid(method):
        """Make sure that TSNE can approximately recover a uniform 2D grid
    
        Due to ties in distances between point in X_2d_grid, this test is platform
        dependent for ``method='barnes_hut'`` due to numerical imprecision.
    
        Also, t-SNE is not assured to converge to the right solution because bad
        initialization can lead to convergence to bad local minimum (the
        optimization problem is non-convex). To avoid breaking the test too often,
        we re-run t-SNE from the final point when the convergence is not good
        enough.
        """
        seeds = [0, 1, 2]
        n_iter = 500
        for seed in seeds:
            tsne = TSNE(n_components=2, init='random', random_state=seed,
                        perplexity=20, n_iter=n_iter, method=method)
            Y = tsne.fit_transform(X_2d_grid)
    
            try_name = "{}_{}".format(method, seed)
            try:
                assert_uniform_grid(Y, try_name)
            except AssertionError:
                # If the test fails a first time, re-run with init=Y to see if
                # this was caused by a bad initialization. Note that this will
                # also run an early_exaggeration step.
                try_name += ":rerun"
                tsne.init = Y
                Y = tsne.fit_transform(X_2d_grid)
>               assert_uniform_grid(Y, try_name)

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.8/site-packages/sklearn/manifold/tests/test_t_sne.py:792: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

Y = array([[-18.169476  ,   6.0802336 ],
       [-18.278513  ,   2.8822129 ],
       [-18.671782  ,  -0.4646889 ],
       ...[ 22.550077  ,  19.698557  ],
       [ 21.399723  ,  22.933178  ],
       [ 16.22136   ,  28.22955   ]], dtype=float32)
try_name = 'barnes_hut_1:rerun'

    def assert_uniform_grid(Y, try_name=None):
        # Ensure that the resulting embedding leads to approximately
        # uniformly spaced points: the distance to the closest neighbors
        # should be non-zero and approximately constant.
        nn = NearestNeighbors(n_neighbors=1).fit(Y)
        dist_to_nn = nn.kneighbors(return_distance=True)[0].ravel()
        assert dist_to_nn.min() > 0.1
    
        smallest_to_mean = dist_to_nn.min() / np.mean(dist_to_nn)
        largest_to_mean = dist_to_nn.max() / np.mean(dist_to_nn)
    
        assert smallest_to_mean > .5, try_name
>       assert largest_to_mean < 2, try_name
E       AssertionError: barnes_hut_1:rerun
E       assert 2.145051767903112 < 2

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.8/site-packages/sklearn/manifold/tests/test_t_sne.py:807: AssertionError
```


```

The following NEW packages will be INSTALLED:

    apipkg:                      1.5-py_0               conda-forge
    attrs:                       19.3.0-py_0            conda-forge
    binutils_impl_linux-aarch64: 2.29.1-hc862510_0      c4aarch64  
    ca-certificates:             2019.11.28-hecc5488_0  conda-forge
    certifi:                     2019.11.28-py38_0      conda-forge
    cython:                      0.29.14-py38he1b5a44_0 conda-forge
    execnet:                     1.7.1-py_0             conda-forge
    importlib_metadata:          1.2.0-py38_0           conda-forge
    joblib:                      0.14.0-py_0            conda-forge
    libblas:                     3.8.0-14_openblas      conda-forge
    libcblas:                    3.8.0-14_openblas      conda-forge
    libffi:                      3.2.1-h4c5d2ac_1006    conda-forge
    libgcc-ng:                   7.3.0-h5c90dd9_0       c4aarch64  
    libgfortran-ng:              7.3.0-h6bc79d0_0       c4aarch64  
    liblapack:                   3.8.0-14_openblas      conda-forge
    libopenblas:                 0.3.7-h5ec1e0e_4       conda-forge
    libstdcxx-ng:                7.3.0-h5c90dd9_0       c4aarch64  
    more-itertools:              8.0.2-py_0             conda-forge
    ncurses:                     6.1-hf484d3e_1002      conda-forge
    numpy:                       1.17.3-py38h91f3968_0  conda-forge
    openssl:                     1.1.1d-h516909a_0      conda-forge
    packaging:                   19.2-py_0              conda-forge
    pluggy:                      0.13.1-py38_0          conda-forge
    py:                          1.8.0-py_0             conda-forge
    pyparsing:                   2.4.5-py_0             conda-forge
    pytest:                      5.3.1-py38_0           conda-forge
    pytest-forked:               1.1.2-py_0             conda-forge
    pytest-sugar:                0.9.2-py_0             conda-forge
    pytest-timeout:              1.3.3-py_0             conda-forge
    pytest-xdist:                1.30.0-py_0            conda-forge
    python:                      3.8.0-heaf0f07_5       conda-forge
    readline:                    8.0-h75b48e3_0         conda-forge
    scikit-learn:                0.22-py38h1971d64_0    local      
    scipy:                       1.3.2-py38hb5cb654_0   conda-forge
    setuptools:                  42.0.2-py38_0          conda-forge
    six:                         1.13.0-py38_0          conda-forge
    sqlite:                      3.30.1-h283c62a_0      conda-forge
    termcolor:                   1.1.0-py_2             conda-forge
    tk:                          8.6.10-hed695b0_0      conda-forge
    wcwidth:                     0.1.7-py_1             conda-forge
    xz:                          5.2.4-hda93590_1001    conda-forge
    zipp:                        0.6.0-py_0             conda-forge
    zlib:                        1.2.11-h516909a_1006   conda-forge

```

https://cloud.drone.io/conda-forge/scikit-learn-feedstock/32/3/2


user: hmaarrfk
labels: arch:arm: ARM related issues
module:test-suite: everything related to our tests
issue_15815: state: open
title: Mod: Modify silhouette_score(), calinski_harabasz_score() and davies_…
body: Modify `silhouette_score()`, `calinski_harabasz_score()` and `Davies-Bouldin Index()` to be compatible with GridSearchCv() when the numbers of clusters is equal to 1 or equal to n_samples.

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
This pull request is to fix this issues https://github.com/scikit-learn/scikit-learn/issues/15717


#### What does this implement/fix? Explain your changes.
For `silhouette_score(X, y)`: instead of raise an `ValueError` when the len(classes_) = 1 or len(X), I return the min possible value = -1.

For `calinski_harabasz_score(X, y)`:  instead of raise an `ValueError` when the len(classes_) = 1 or len(X), I return the min possible value = 0.

For `Davies-Bouldin Index()`: instead of raise an `ValueError` when the len(classes_) = 1 or len(X), I return the max possible value = np.inf. This scorer does not follow the convention that higher return values are better than lower return values used along scikit-learn. It follows the opposite for that I return np.inf (could be fixed to a very big value)

I change few lines in sklearn/metrics/clusters/tests/test_unsupervised.py and passed all the tests.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: rafikg
issue_15808: state: open
title: add method 'transform' for SpectralEmbedding
body: For `SpectralEmbedding`, is there any plan to add the method `transform`? This is related to the old post #2887
user: huidongchen
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_15801: state: open
title: RFC Support for Some Developer Utilities
body: On the [Utilities for Developers](https://scikit-learn.org/stable/developers/utilities.html) page, it states:

> **Warning:** These utilities are meant to be used internally within the scikit-learn package. They are not guaranteed to be stable between versions of scikit-learn. Backports, in particular, will be removed as the scikit-learn dependencies evolve.

If we want to provide utilities to support third-party estimators, we should treat some of these utilities as "first class" citizens.

For example `safe_indexing` would be extremely useful for third parties that want to support DataFrames as input. Currently, the options for third-party developers is to build their own "safe_indexing" or depend on our private version which may not be stable.

Another example is https://github.com/scikit-learn/enhancement_proposals/pull/22, which defines a `n_features_in_` contract where we will internally use private methods to cohere with the contract. Third-party estimators would need to build their own methods or functions to work with the SLEP.

**TLDR:** Now that much of the utilities are "private", we can make deliberate decisions about what utilities should be public and supported by us. This would mean deprecation cycles, etc. If we support some of the `utils` module, it will make it easier to build estimators, which will enrich the ecosystem of scikit-learn compatible estimators.

CC @scikit-learn/core-devs 
user: thomasjpfan
labels: Hard: Hard level of difficulty
Needs Decision - API: Requires decision regarding API
issue_15796: state: open
title: META OHE / OrdinalEncoder: NaN support, unfrequent cat. and pd.Categorical
body: There have been 3 significant improvements proposed for `OneHotEncoder` (and to a lesser extent `OrdinalEncoder`), often with an associated PR,
 - NaN handling (issue https://github.com/scikit-learn/scikit-learn/issues/11996, PR https://github.com/scikit-learn/scikit-learn/pull/13028)
 - support of `pd.Categorical` dtype (issue https://github.com/scikit-learn/scikit-learn/issues/14953, PR https://github.com/scikit-learn/scikit-learn/pull/15396)
 - handling of infrequent categories (issue https://github.com/scikit-learn/scikit-learn/issues/12153, PR https://github.com/scikit-learn/scikit-learn/pull/13833)

the goal of this issue to have a high level agreement on the desired solution, that it is consistent/compatible for different available encoders, or encoders that we may want to add in the near future (e.g. target encoder, https://github.com/scikit-learn/scikit-learn/issues/5853). Some of the possible solutions for the above 3 features are mutually exclusive. Also putting aside aside backward compatibility constraints for a start, what default options we would want ideally.

I have not followed in detail all past discussions about encoders (in particular about ordering concerns https://github.com/scikit-learn/scikit-learn/pull/15050). Following are some of the observations / open questions I have, please add more if I missed something/link with existing comments.

**NaN support**

 - Mainly if we want to implement support directly or suggest to use an imputer for the pre-processing step (as one can do now).

**pd.Categorical support**
 - Do we say that `categories='dtype'` would make OHE categories match categories in the dtype 
? Including the ordering? But then, this means only at fit since, for transform, the test set could have unknown categories. 
 - Actually, if one does a train test split, some categories from the dtype can be missing from the train set as well. Do we then create a column with 0s in `fit_transform`, or disregard this column breaking the assumption of conforming to dtype categories? Solution proposed by @thomasjpfan in https://github.com/scikit-learn/scikit-learn/pull/15396#issuecomment-552065493
 - Finally, if we do not conform to the dtype categories, and only use `pd.Categorical` for computational efficiency internally, what is the point of defining a `categories='dtype'` in the first place (or warning that categories order doesn't match the order of categories in dtype https://github.com/scikit-learn/scikit-learn/pull/15050).
 
**Infrequent categories**

Overall the plan seems fairly clear in https://github.com/scikit-learn/scikit-learn/issues/12153#issuecomment-561774199

**NaN & pd.Categorical**

 - Handing `NaN` as a separate category, means we are no longer using purely the categories from the dtype (even with `dtype='category'`) which can be fine as long as we agree on it.
 - Another possibility is to implement a say `CategoricalPreprocessor` to add `NaN` as a new dtype category in a separate preprocessing step,
   ```py
   df[column].cat.add_categories("NaN", inplace=True)
   df[column].fillna("NaN", inplace=True)
   ```
   which can make interpretation simpler. Say `df[column].value_counts()` would then show NaN properly and one might want to do it for exploratory analysis in any case. 
 
**NaN & infrequent categories**

- Do infrequent categories rules apply to NaN or is it always a separate category (even if passes the infrequent criteria)?

**pd.Categorical and infrequent categories**

- Similarly we could consider a preprocessor that would add `infrequent` as a category to dtype, instead of doing that internally in encoders. Say to evaluate how many infrequent elements one has, I find that doing (approximately),
  ```py
  ohe = OneHotEncoder(categories='dtype', min_frequency=5)
  X = ohe.fit_transform(df['col'])
  unfrequent_idx = ohe.get_feature_names().tolist().find("infrequent")
  print(X.sum(axis=0).A1[unfrequent_idx])
  ```
  very awkward as opposed to,
  ```py
  df['col'].value_counts()["infrequent"]
  ```
  where it was properly added to `df['col'].cat.categories` previously and we are using all the nice features of `pd.Categorical`. 

So there is some tension here between adding these features to scikit-learn and keeping exploratory analysis with pandas user friendly (and not asking users to implement the same thing twice).

Given the complexity of this interaction, maybe separating "Imputer + Unfrequent categories conversion with pd.Categorical support" and "OneHot, Ordinal, Target etc encoder" into 2 or 3 estimators might be easier to understand? Not sure about usability though. The alternative that would mean we also plan to add these features (and enforce consistency) for any future encoder.

cc @thomasjpfan @NicolasHug @glemaitre @jorisvandenbossche  @amueller @jnothman @ogrisel 
user: rth
issue_15783: state: open
title: StratifiedKFold yields slightly different class label distributions (prior to 0.22 different train and test sizes)
body: #### Description
Unfortunately, probably due to the stratification balancing, the train and test size can vary between iteration.  This is pretty highly undesirable and it should always rebalance to same sizes.  Otherwise combining all the folds and using efficient vectorized operations to do computations in a matrix with an extra dimension is not easily possible.

This honestly feels like a poor stratification algorithm.  At worst this is a flat out bug.  At best, its a mode of operation, and a Boolean flag should be added such as something like `force_exact_division=True`.  Of course, in some cases it is impossible to balance the classes, but even in those cases, it could still prefer to guarantee equal distribution, rather than guarantee equal class sizes since we are talking about only fractional remainders here - and the class sizes are not going to be perfect percentage wise balanced anyway as it can be impossible for example if only 1 value belongs to a class.  This is surprisingly awkward functionality.

#### Steps/Code to Reproduce
```
import numpy as np
from sklearn.model_selection import StratifiedKFold
kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
X, y = np.arange(7590)[:,np.newaxis], np.random.RandomState(0).randint(0, 3, 7590)
for (train, test) in kfold.split(X, y):
    print(len(train), len(test))
kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
for (train, test) in kfold.split(X, y):
    assert(len(test) == 7590/5)
    assert(len(train) == 7590*4/5)
```

#### Expected Results
```
6072 1518
6072 1518
6072 1518
6072 1518
6072 1518
```

#### Actual Results
```
6071 1519
6071 1519
6072 1518
6072 1518
6074 1516
Traceback (most recent call last):

  File "<ipython-input-106-1c41a73d61b2>", line 8, in <module>
    assert(len(test) == 7590/5)

AssertionError
```

Seems to always be in order from `min(len(train))` to `max(len(train))`

#### Versions
Windows-10-10.0.18362-SP0
Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]
NumPy 1.17.3
SciPy 1.3.1
Scikit-Learn 0.21.3
user: GregoryMorse
labels: Needs Decision: Requires decision
issue_15781: state: open
title: Using ColumnTransformer when not all specified columns present in data
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
One needs to specify columns to which certain transformers are to be applied. However, when a dataset fed is missing one of the specified columns, an error is raised. I would like to still be able to use the ColumnTransformer, even when some of the specified columns are not present in the data. Is this possible?

#### Steps/Code to Reproduce
```python
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestRegressor
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer, SimpleImputer
from sklearn.pipeline import Pipeline
import sklearn.datasets
import pandas as pd

X, y = sklearn.datasets.fetch_openml('iris', 1, return_X_y=True)
X = pd.DataFrame(X, columns = ["one", "two", "three", "four"])

numeric_features = ["one", "two", "three", "four", "five"]

numeric_transformer = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="constant", fill_value=0)),
    ]
)

preprocessor = ColumnTransformer(
    transformers=[
        ("num", numeric_transformer, numeric_features),
    ],
    remainder="drop",
    sparse_threshold=0,
)

preprocessor.fit_transform(X)
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
A fitted df/array 

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
```python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/sklearn/compose/_column_transformer.py", line 474, in fit_transform
    self._validate_remainder(X)
  File "/usr/lib/python3.8/site-packages/sklearn/compose/_column_transformer.py", line 315, in _validate_remainder
    cols.extend(_get_column_indices(X, columns))
  File "/usr/lib/python3.8/site-packages/sklearn/compose/_column_transformer.py", line 703, in _get_column_indices
    return [all_columns.index(col) for col in columns]
  File "/usr/lib/python3.8/site-packages/sklearn/compose/_column_transformer.py", line 703, in <listcomp>
    return [all_columns.index(col) for col in columns]
ValueError: 'five' is not in list
```

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
``` python
System:
    python: 3.8.0 (default, Oct 23 2019, 18:51:26)  [GCC 9.2.0]
executable: /bin/python
   machine: Linux-4.9.130-xxxx-std-ipv6-64-x86_64-with-glibc2.2.5

Python deps:
       pip: 19.2.3
setuptools: 41.6.0
   sklearn: 0.21.3
     numpy: 1.17.4
     scipy: 1.3.1
    Cython: 0.29.14
    pandas: 0.25.3
```
<!-- Thanks for contributing! -->
user: larskouwenhoven
issue_15766: state: open
title: Acknowledge bug reports in what's new?
body: We acknowledge bug-fixes in the whats new logs, bug not bug reports. Maybe we could do that?

It does take time and commitment to write a bug report (esp a good one), and it would be a way to show our community that we appreciate that kind of feedback.

e.g. `:pr:1234 by X, bug report :issue:4321 by Y`
user: NicolasHug
issue_15755: state: open
title: What's the recommended approach for building a complete data frame (feature values + names) after using ColumTransformer/ FeatureUnion?
body: I posted a question on StackOverflow but unfortunately without any reply. Could you guys please take a quick look at this end recommend something? I have a feeling there's more people looking for a similar, clear answer which I haven't been able to find anywhere so your input would be very much appreciated!

https://stackoverflow.com/questions/58989517/whats-the-recommended-approach-for-building-a-complete-data-frame-feature-valu

Best
Konrad
user: konradsemsch
issue_15749: state: open
title: Cannot transform unknown length data/chunked data with a Pipeline (no ChunkedTransformer/ChunkedEstimator/ChunkedClassifier)
body: #### Description

So you have a more than can fit in memory amount of data in separate files on the disk.  There are frameworks like dask which you could use to compile them, but that's a huge amount of data copying for little benefit.  The total data length could be calculated but there is no real need to do that either.  There is no way to perform feature selection in chunks with partial fitting followed by concatenated transforming and then fit post transformation on a reasonable size of data using a Pipeline.

Furthermore the transformers may be used for certain features only.  So 10000 features becomes 10 transformers with 1000 features each, and the best 5 features of each transform is used, resulting in 50 features.  This division of features into the transformers is also important due to memory constraints.  Of course ColumnTransformer solves the task of dividing columns among transformers and choosing the best amongst those.

The assumption is enough memory is available for 50 features over all the data.  But 1000 features over all the data will not fit in memory.  I see no way a Pipeline can support chunked data/unknown length/load on demand data.  Basically you want to partial_fit the data, then concatenate while transforming the data then move on to the next stage of the pipeline which could have more transformers and then an estimator.  Only the very first stage transformer of the pipeline should need this special operation.  However in some cases perhaps the intermediate phases may need to be cached and redone with the same strategy.

#### Steps/Code to Reproduce

Say there are 1000 files with various rows of data (1000-10000), and 50000 features each.  Feature selection will reduce these 50000 features down to a mere 50 by choosing the best.  Its assumed the data will fit into memory despite the numbers used after feature selection but not before so a full fit can be done.

What you want to do:
Stage 1: `partial_fit` over and over on the transformers for each chunk of data
Stage 2: `transform` for each chunk of data with all the transformers which will do the feature selection
Stage 3: concatenate the results
Stage 4: `fit` on the estimator

This would seem to be an incredibly common idea for data sets which are too large, but feature extraction can reduce them down to a reasonable size.

#### Solutions
Add **ChunkedTransformer** to scikit learn.  This will perform partial_fit in chunks then transform in chunks and compile the result.  I suppose **ChunkedEstimator** and **ChunkedClassifier** could also provide wrappers where `partial_fit` is available or for overly large data in `predict`.  The input `X` format would need to be slightly different as an `iterator` of arrays rather than a single array but the output would be the same.  Caching in a Pipeline is already supported so intermediate caching and loading is at least possible.  These are simple classes to implement as they merely wrap an underlying Transformer/Estimator/Classifier and feed the partial_fit or transform or predict functions from an iterable and compile results.  Unfortunately there are a good deal of places where the `X` parameter is assumed to not be of a generic iterable format and might take some refactoring in those places to deal with it such as in GridSearchCV.

Another idea is that sklearn support callback functions for `X` in models which can load the data in chunks via iteration.  But that's probably too much of a massive change.  And the `fit_transform` would still need to enumerate the data twice using that.  Perhaps a class can do this overriding the various `__getitem__` methods to provide a virtualized view of the data.  Nonetheless the length is not known so it would have to still fetch it in chunks.
Another is a partial_fit method which can be called repeated only the pipeline.  Then a `transform_fit` method automatically will concatenate the results of the transform and fit on the estimator.  These 2 ideas are basically dead ends since they do not provide the granularity of flexibility for multiple stages of transformers in the pipeline.  The first idea of adding **Chunked** wrappers is the only real solution.

Is there something I am missing or is this incredibly obvious task not possible with a pipeline?  I would like to discretize my data in the transformers, and also have the transformer use a scoring algorithm to choose the best features when discretized.  Then finally the fitting can occur.  But I would like to do it on a good deal of data.  Of course another solution without a pipeline:
1. `partial_fit` in a loop until data exhausted.
2. `transform` in a loop with concatenation until data exhausted
3. pass to estimator with `fit`

But now the benefit of using something like GridSearchCV to tune the feature selection and model together is missing.
user: GregoryMorse
labels: Needs Decision: Requires decision
issue_15737: state: open
title: [MRG] metrics.most_confused_classes (issue #15696)
body: 
#### Reference Issues/PRs

Resolves #15696 Generate the pairs of most confused classes in multiclass classification

#### What does this implement/fix? Explain your changes.

This pull request implements the function most_confused_classes in the metrics module. This function computes the classes have been confused the most often by the classifier. Applies for both binary and multiclass classification problems.
It returns a np.ndarray with each row containing: [class_1, class_2, count] where count denotes the number of times class_1 was misclassified as class_2.

*Example:*

    >>> from sklearn.metrics import most_confused_classes
    >>> y_true = [2, 0, 1, 2, 1, 2]
    >>> y_pred = [0, 0, 2, 0, 1, 1]
    >>> most_confused_classes(y_true, y_pred)
*will return:*
```
    array([[2, 0, 2],
           [2, 1, 1],
           [1, 2, 1]], dtype=int64)
```

#### Any other comments?

This function was suggested by @regstrtn in the issue linked earlier.
I found this issue and thought that it will be a good first issue for me to learn the scikit-learn contribution workflow, so I worked on it.
I don't know if this function has the potential to be used enough to make it to scikit-learn, but it was fun working on it!
Please let me know how I could potentially improve this.
user: pleydier
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_15727: state: open
title: Promote SKLEARN_SEED for consideration in __init__.py itself
body: Inspired by https://github.com/scikit-learn/enhancement_proposals/pull/24 I decided to look into either it is possible to seed RNG used by sklearn "globally". Apparently it was even me who, after a heated discussion, added treatment of `SKLEARN_SEED` environment variable within `setup_module()` fixture back in 2012 https://github.com/scikit-learn/scikit-learn/commit/4915d4dcd93a1051f4bdf933987943118d02c9a7 to provide means for reproducible testing.   
AFAIK there is no generic treatment of `SKLEARN_SEED` or any other env variable as to define a starting point of RNG for an arbitrary script which (directly or indirectly) uses sklearn.  It would be useful for the cases where there is a script which uses scikit-learn functionality and has no explicit seeding handling built-in.  Setting the seed via env variable provides a chance to provide reproducible results if I re-run that script with the same env variable value without doing any modifications to the script or any underlying library which actually interfaces to scikit-learn and/or other libraries.
This is a strategy we used in PyMVPA and started to collect similar cases (see https://github.com/ReproNim/reproseed/blob/master/reproseed.sh#L24 for the short for now list) with the hope to be able to seed all relevant tools once in case when it is necessary to make some issue or result reproducible.
I wondered if there would be any interest to pursue this direction.
user: yarikoptic
issue_15724: state: open
title: Average path length in iForest is inaccurate for small sizes
body: #### Description
The routine `_average_path_length()` for isolation forests (sklearn.ensemble.IsolationForest) gives quite inaccurate results for small sizes (`n_samples_leaf`), which propagates to the anomaly scores. For `n_samples_leaf=3` the error is around 30%. This is due to the fact that a very rough approximation mentioned in the original paper is used, which only reaches reasonable accuracy for very large numbers. Therefore I would suggest to modify the current routine as follows with a combination of a lookup table for small values and an improved asymptotic series for large values, which should be accurate to double precision. Please let me know what you think. (This might be considered an improvement on https://github.com/scikit-learn/scikit-learn/pull/13251.)

#### Steps/Code to Fix
```
def _average_path_length_small(n_samples_leaf):
    c = (0.0, 0.0, 1.0, 1.6666666666666666667, 
         2.1666666666666666667, 2.5666666666666666667, 2.9000000000000000000, 3.1857142857142857143, 
         3.4357142857142857143, 3.6579365079365079365, 3.8579365079365079365, 4.0397546897546897547, 
         4.2064213564213564214, 4.3602675102675102675, 4.5031246531246531247, 4.6364579864579864580, 
         4.7614579864579864580, 4.8791050452815158698, 4.9902161563926269809, 5.0954793142873638230, 
         5.1954793142873638230, 5.2907174095254590611, 5.3816265004345499702, 5.4685830221736804049, 
         5.5519163555070137383, 5.6319163555070137383, 5.7088394324300906613, 5.7829135065041647354, 
         5.8543420779327361640, 5.9233075951741154743, 5.9899742618407821410, 6.0544903908730402055, 
         6.1169903908730402055, 6.1775964514791008116, 6.2364199808908655175, 6.2935628380337226603, 
         6.3491183935892782159, 6.4031724476433322699, 6.4558040265907006910, 6.5070860778727519730, 
         6.5570860778727519730, 6.6058665656776300218, 6.6534856132966776409, 6.6999972412036543850, 
         6.7454517866581998396, 6.7898962311026442840, 6.8333744919722095014, 6.8759276834615712036, 
         6.9175943501282378702, 6.9584106766588501151, 6.9984106766588501151, 7.0376263629333599190)
    #return np.array(c)[n_samples_leaf]
    return np.array(c)[np.maximum(0, n_samples_leaf)]

def _average_path_length(n_samples_leaf):
    """
    The average path length in a n_samples iTree, which is equal to
    the average path length of an unsuccessful BST search since the
    latter has the same structure as an isolation tree.
    Parameters
    ----------
    n_samples_leaf : array-like, shape (n_samples,).
        The number of training samples in each test sample leaf, for
        each estimators.

    Returns
    -------
    average_path_length : array, same shape as n_samples_leaf
    """

    n_samples_leaf = check_array(n_samples_leaf, ensure_2d=False)

    n_samples_leaf_shape = n_samples_leaf.shape
    n_samples_leaf = n_samples_leaf.reshape((1, -1))
    average_path_length = np.zeros(n_samples_leaf.shape)

    mask_small = n_samples_leaf < 52
    not_mask = ~mask_small

    average_path_length[mask_small] = _average_path_length_small(n_samples_leaf[mask_small])

    tmp = 1.0/np.square(n_samples_leaf[not_mask])
    average_path_length[not_mask] = (
        2.0 * (np.log(n_samples_leaf[not_mask]) - 1.0 + np.euler_gamma)
        + 1.0/n_samples_leaf[not_mask] - tmp*(1.0/6 - tmp*(1.0/60 - tmp/126))
    )

    return average_path_length.reshape(n_samples_leaf_shape)
```

#### Versions
System:
    python: 3.8.0 (default, Oct 23 2019, 18:51:26)  [GCC 9.2.0]
executable: /usr/bin/python
   machine: Linux-5.3.13-arch1-1-x86_64-with-glibc2.2.5

Python deps:
       pip: 19.2.3
setuptools: 41.6.0
   sklearn: 0.21.3
     numpy: 1.17.4
     scipy: 1.3.1
    Cython: 0.29.14
    pandas: 0.25.3


<!-- Thanks for contributing! -->

user: Konrad0
issue_15717: state: open
title: Silhouette score is not defined when the estimator return only 1 cluster
body: https://github.com/scikit-learn/scikit-learn/blob/4f97facc3a992c6e2459c3da86c9d69b0688d5ab/sklearn/metrics/cluster/_unsupervised.py#L38
I am wondering why the silhouette score is not defined when the estimator return only 1 cluster. In this case, if we want to do Hyperparameter tuning (GridSearch) based on the silhouette score, it will crash if the estimator return one cluster. In this case, I am wondering why we don't return the worst case (silhouette score = -1)? 
user: rafikg
issue_15710: state: open
title: [WIP] Add support for Histogram-Like Multi-Labeled Encodings with Associated Values
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
This allows the user to create multi-labeled encodings for dict-like objects with classes and associated values. My use-case is encoding a histogram with string-like keys. MultiLabelBinarizer does not support dictionaries with fuzzy values, only discrete `{0,1}` sets. 

Examples:
```
    mlh = MultiLabelHistogram()
    y = [{1: 5.5, 2: -3.0}, {3: 999}]
    Y = np.array([[5.5, -3., 0.],
                  [0., 0., 999.]], dtype=np.float32)
    assert_allclose(mlh.fit_transform(y), Y)
```
```
    mlh = MultiLabelHistogram()
    y = [{'sci-fi': -2.0, 'thriller': 5.0}, {'comedy': 0.1}]
    Y = np.array([[0., -2., 5.],
                  [0.1, 0., 0.]], dtype=np.float32)
    assert_allclose(mlh.fit_transform(y), Y)
```

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: chuckhacker
issue_15702: state: open
title: [MRG] ENH Improves on the plotting api by storing weak ref to display in axes
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Resolves https://github.com/scikit-learn/scikit-learn/issues/15581 (better)


#### What does this implement/fix? Explain your changes.
After a discussion with @tacaswell, it was concluded that adding attribute to the axes would be solution to figuring out

This allows use cases in #15581 to work:

```py
fig, ax = plt.subplots(figsize=(10, 6))
tree_disp = plot_partial_dependence(tree, X, ["LSTAT"], ax=ax)
mlp_disp = plot_partial_dependence(mlp, X, ["LSTAT"], ax=ax,
                                   line_kw={"c": "red"})
```

#### Any other comments?
Storing the display object in the axes so it can be referenced later is a small hack, but it improves the API quite a bit.

CC @tacaswell @amueller @NicolasHug @glemaitre @qinhanmin2014 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
issue_15696: state: open
title: Generate the pairs of most confused classes in multiclass classification
body: Use case: 
Multiclass classification

I think it would be helpful if the user could see the pairs of classes that are being confused most often in a multiclass classification, by calling a method `most_confused_classes(y_true, y_pred)`. 
The method should return a matrix of size = `[n_pairs, 3]`. 
Each row will contain `<class1, class2, count>` where count denotes the number of times class1 was misclassified as class2.


user: regstrtn
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_15667: state: open
title: CCA sets condition in pinv2 to match with scipy < 1.3
body: The fix in https://github.com/scikit-learn/scikit-learn/pull/15661 sets the condition in `scipy.linalg.pinv2` to match with scipy < 1.3. It would be good to update the algorithm to not need to do this.

We have ran into a similar issue in https://github.com/scikit-learn/scikit-learn/pull/14067 where we vendor the previous version of `scipy.linalg.pinvh` to keep the old condition.
user: thomasjpfan
issue_15657: state: open
title: RFC Sample weight invariance properties
body: *This can wait after the release.*

A discussion happened in the GLM PR https://github.com/scikit-learn/scikit-learn/pull/14300 about what properties we would like `sample_weight` to have. 

First, a short side comment about 3 ways simple weights (`s_i`) are currently used in loss functions with regularized generalized linear models in scikit-learn (as far as I understand),

- <img src="https://user-images.githubusercontent.com/630936/69103411-9ff79900-0a65-11ea-8cf5-03a183a3a7bd.png" width="40%" /> 
    
   For instance: `Ridge` (also `LogisticRegression` where `C=1/α`)
 
- <img src="https://user-images.githubusercontent.com/630936/69103589-17c5c380-0a66-11ea-9bb2-bd6654c7b8ba.png" width="45%" />
  
    For instance: `SGDClassifier`? (maybe `Lasso`, `ElasticNet` once they are added?)

 - <img src="https://user-images.githubusercontent.com/630936/69103706-64110380-0a66-11ea-96cb-35fc8b4035dc.png" width="47%" />

    For instance, currently proposed in the GLM PR for `PoissonRegressor` etc

For sample weight it's useful to think in term of invariant properties, as they can be directly expressed in common tests. For instance,

  1. checking that zero sample weight is equivalent to ignoring samples in https://github.com/scikit-learn/scikit-learn/pull/15015 (replaced by #17176) helped discovering a number of issues.
    ~~All of the above formulations should verify this.~~ It is verified only by `L_1a` and `L_2b`.

Similarly, paraphrasing https://github.com/scikit-learn/scikit-learn/pull/14300#issuecomment-543177937 other properties we might want to enforce, are,

  2. multiplying some sample weight by `N` is equivalent to repeating the corresponding samples N times.
  It is verified only by `L_1a` and `L_2b`.
  **Example:** For `L_2a` setting all weights to 2, is equivalent to having 2x more samples only if `α = α / 2`.
  
  3. Finally, that scaling sample weight has no effect. This is only verified by `L_2b`. For both `L_1a` and `L_2a` multiplying all samples weights by `k` is equivalent to setting `α = α / k`.
     
     This one is more controversial. Against enforcing this,
      - there are arguments of keeping a meaning for business metrics (e.g. https://github.com/scikit-learn/scikit-learn/issues/15651)
    
     in favor,
      - that we don't want a coupling between using samples weight and regularization. 
        **Example:** Say one has a model without sample weights, and one wants to see if applying samples weights (imbalanced dataset, sample uncertainty, etc) improves it. Without this property it's difficult to conclude: is the evaluation metric better with sample weights, due to those, or simply because we now have a better regularized model? One has to simultaneously consider these two factors.
        
Whether we want/need consistency between the use of sample weight in metrics in estimators is another question. I'm not convinced we do, since in most cases estimators don't care about the global scaling of the loss function, and  these formulations are equivalent up to a scaling of the regularization parameter. So maybe using the `L_1a` equivalent expression in metrics could be fine. 
        
In any case, we need to decide the behavior we want. This is a blocker for,
 - Poisson, Gamma and Tweedie Regression https://github.com/scikit-learn/scikit-learn/pull/14300
 - adding sample weights in `ElasticNet` and Lasso https://github.com/scikit-learn/scikit-learn/pull/15436
 - other tests for sample weights consistency in linear models by @lorentzenchr in https://github.com/scikit-learn/scikit-learn/pull/15554
 
**Note:** Ridge actually seem to have a different sample weight behavior for dense and sparse as reported in https://github.com/scikit-learn/scikit-learn/issues/15438

@agramfort 's option on this can be found in https://github.com/scikit-learn/scikit-learn/issues/15651#issuecomment-555210612 (if I understood correctly).

Please correct if I missed something (this could also use a more in depth review of how it is done in other libraries).
user: rth
issue_15651: state: open
title: RFC Semantic of sample_weight in regression metrics
body: Several metrics in scikit-learn are using `np.average(..., weight=sample_weight)` under the hood:

* `mean_absolute_error`
* `mean_squared_error`
* `explained_variance_error`
* `r2_score`
* `mean_tweedie_deviance`

When computing the average, `numpy` will divide by the sum of the weights. Is it the intended behaviour. For instance:

```python
sample_weight = [1, 2, 3, 4]
```

and

```python
sample_weight = [1/10, 2/10, 3/10, 4/10]
```

will lead to the same error/score. Dividing by the sum of the weight will also remove any meaning about the use of a unit (if `sample_weight` is related to a business unit for instance).

So I was wondering if we should multiply the average by the sum of the weight or not. 
user: glemaitre
issue_15649: state: open
title: NMF with some topics/meta-features held constant
body: In cases where you know in advance some of the topics/meta-features in your data, and you still wish to run NMF to find additional topics/meta-features (W matrix) and the topics/meta-features importance per sample (H matrix), you basically need to optimize over the same NMF objective, but with some columns of the W matrix held constant throughout the optimization process. In the extreme case where all topics/meta-features are known in advance, this reduces to optimization over H alone. 
This is a feature request on work I'm currently pursuing as part of a research project. I'm opening this feature request in case someone is already working on something similar, or if I totally missed something that implements this functionality. 


user: akhiadber
issue_15648: state: open
title: [MRG] FIX and ENH in _RidgeGCV
body: Fixes #4667 Fixes #4790 Fixes #13998 Fixes #15182 Fixes #15183

This PR focus on RidgeCV
- add best_score_ to RidgeCV, in order to write tests (#4667 #4790)
- do not store all cv values when store_cv_values == False (#15182 #15183 )
- when scoring = None
    - correct the errors (previously they are erroneously weighted)
    - correct the scores (previously they are erroneously weighted)
- when scoring != None
    - correct the predictions (previously they are erroneously weighted and scaled)
    - correct the scores (previously they are erroneously weighted and scaled)
    - handle multioutput correctly when calculating the scores

TODO in this PR: update the doc, update what's new

TODO: issues regarding RidgeClassifierCV
ping @glemaitre feel free to edit or push
user: qinhanmin2014
issue_15647: state: open
title: Gaussian Mixture - Implementing weighted sample
body: I implemented a version of Gaussian Mixture for weighted data (fixed values).
The mathematical framework can be found here (part III):
https://arxiv.org/abs/1509.01509
So if a sample has a weight _w_, it means the sample is observed _w_ times. 

mixture/base.py and mixture/gaussian_mixture are modified by:
- adding the sample weights to the EM algorithm (Eq 6, 8, 9, 10 of the paper)
- adding the sample weights to the log likelihood
- adding a sample_weight argument to fit, predict, score... methods. 

I followed the same logic than the KMeans class for the user interface (with sample_weight as an optional keyword) and use _check_sample_weight.

Is it worth to open a PR or this feature does not have much interest for the official release?
user: dupuisIRT
issue_15641: state: open
title: Inconsistency between partial_dependence and plot_partial_dependence
body: Something annoying:
```
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.inspection import plot_partial_dependence

X, y = load_breast_cancer(return_X_y=True)
clf = LogisticRegression()
clf.fit(X, y)

partial_dependence(clf, X, features=[(0, 1), 2])
# raise an error
plot_partial_dependence(clf, X, features=[(0, 1), 2])
# no error
```
IMO plot_XXX should plot the output of XXX, it should not provide additional features.
Related issue: #15625
user: qinhanmin2014
issue_15640: state: open
title: Inconsistency between LDA and QDA covariance estimation
body: #### Description

LDA and QDA are inconsistent in how they estimate the class wise covariance matrix.

For QDA the relevant line of code is:

https://github.com/scikit-learn/scikit-learn/blob/1495f69242646d239d89a5713982946b8ffcf9d9/sklearn/discriminant_analysis.py#L694

i.e. the normalization `1/(N-1)` is used, whereas LDA calls 

https://github.com/scikit-learn/scikit-learn/blob/1495f69242646d239d89a5713982946b8ffcf9d9/sklearn/covariance/empirical_covariance_.py#L49

which by default uses `np.cov` with `bias=True`, which is equivalent to `ddof=0`, corresponding to the normalization `1/N`

#### Steps/Code to Reproduce

```python
import numpy as np
np.set_printoptions(4)
data = np.array([
  [ 1,  1,  "A"],
  [ 2 , 1 , "A"],
  [ 2 , 2 , "A"],
  [ 1, -1,  "B"], 
  [-1, -1,  "B"],
  [-2, -2,  "B"]])

X = data[:, :-1].astype(float)
Y = data[:,  -1]
classes = np.unique(Y)
pi_A = len(X[Y=="A"])/len(X)
pi_B = len(X[Y=="B"])/len(X)
cov_A  = np.cov(X[Y=="A"], rowvar=False, ddof=0)  
cov_B  = np.cov(X[Y=="B"], rowvar=False, ddof=0)  
cov_A2 = np.cov(X[Y=="A"], rowvar=False, ddof=1)
cov_B2 = np.cov(X[Y=="B"], rowvar=False, ddof=1)
sigmahat = pi_A*cov_A + pi_B*cov_B
sigmahat2 = pi_A*cov_A2 + pi_B*cov_B2

print("**ddof=0**", "cov_A:", cov_A, "cov_B:", cov_B, "sigmahat:", sigmahat,"\n", sep="\n")
print("**ddof=1**", "cov_A:", cov_A2, "cov_B:", cov_B2, "sigmahat:", sigmahat2,"\n", sep="\n")

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
model = LDA(store_covariance=True)
model.fit(X, Y)
print("**LDA**","sigmahat:", model.covariance_,"\n", sep="\n")

from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis as QDA
model = QDA(store_covariance=True)
model.fit(X, Y)
covs=model.covariance_
print("**QDA**", "cov_A:", covs[0], "cov_B:", covs[1],"\n", sep="\n")
```


#### Expected Results

There are two common ways to estimate the joint covariance matrix in LDA:

1. estimate all class covariance individually and aggregate them as
`sigmahat = sum prior(class)*covariance(class)`
2. estimate the common covariance "jointly" (https://en.wikipedia.org/wiki/Pooled_variance)
`sigmahat = 1/(N-num_classes) sum_{c in classes} sum_{x : y=c} (x_i - mu_c)*(x_i -mu_c).T`

The sklearn LDA model seems to do the former by calling

https://github.com/scikit-learn/scikit-learn/blob/1495f69242646d239d89a5713982946b8ffcf9d9/sklearn/discriminant_analysis.py#L96

In this case, both LDA and QDA should estimate the class covariance with he same estimator. `ddof=0` is appropriate, because the model presupposed that the classes are normally distributed and `ddof=0` is the maximum likelihood estimator in this case.

#### Actual Results
 LDA uses `ddof=0` and QDA uses `ddof=1`

#### Versions
```
System:
    python: 3.7.5 (default, Oct 25 2019, 15:51:11)  [GCC 7.3.0]
executable: /home/rscholz/miniconda3/bin/python
   machine: Linux-5.3.0-23-generic-x86_64-with-debian-buster-sid

Python deps:
       pip: 19.3.1
setuptools: 41.6.0.post20191030
   sklearn: 0.21.3
     numpy: 1.17.3
     scipy: 1.3.1
    Cython: None
    pandas: 0.25.2
```

<!-- Thanks for contributing! -->

user: randolf-scholz
issue_15629: state: open
title: Standardized input produces different prediction for decision tree than original input
body: #### Description

Standardizing input should have no effect on decision trees. I fit a `DecisionTreeRegressor` on both the original and scaled (transformed with `StandardScaler`) and used it to make predictions on an unseen dataset. One of the predictions was not equal. 

I believe this might be due to floating point accuracy. It seems the tree stores the decision points as float32, so one of the decisions on the boundary was not made properly.

[Gist is here.](https://gist.github.com/tdpetrou/fc8d93788f7c20f8e2db8ade58b07ecb)

#### Steps/Code to Reproduce

I discovered this issue using the famous Kaggle housing dataset doing cross validation on the second fold. I split the data below setting a `random_state`  for `KFold`

```
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import KFold, cross_val_score
from sklearn.tree import DecisionTreeRegressor

housing = pd.read_csv('http://bit.ly/kagglehousingtrain')
cols = ['GrLivArea', 'GarageArea']
X = housing[cols].values

ss = StandardScaler()
X_scaled = ss.fit_transform(X)
kf = KFold(n_splits=5, shuffle=True, random_state=999)

train_idx, test_idx = list(kf.split(X))[1]
X_train, X_test = X[train_idx], X[test_idx]
X_scaled_train, X_scaled_test = X_scaled[train_idx], X_scaled[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
````

I set a `random_state` for `DecisionTreeRegressor` and train on the original data. I get the predicted values on the test set and print out the score.

```
dtr = DecisionTreeRegressor(max_depth=2, random_state=123)
dtr.fit(X_train, y_train)
y_pred = dtr.predict(X_test)
print(dtr.score(X_test, y_test))
```

The score is **.436518997**. 

I train the model using the scaled data, get the predicted values on the scaled test set and print out the score.

```
dtr.fit(X_scaled_train, y_train)
y_pred_scaled = dtr.predict(X_scaled_test)
print(dtr.score(X_scaled_test, y_test))
```

This time the score is **.43707501** which is slightly different than above. Given that `random_state` was set, these two values should be the same. I verify that one of the predicted values is indeed not the same.

```
filt = y_pred != y_pred_scaled
print(y_pred[filt])
print(y_pred_scaled[filt])
```

**119791** vs **163336**

I output the original and scaled values

```
print(X_test[filt])
print(X_scaled_test[filt])
```
[[1199  530]]
[[-0.60244329  0.26678255]]

1199 and -0.60244329 are each the threshold for one of the decisions in their respective trees. For the original dataset the tree evaluates 1199 <= 1199 as True. In the scaled tree, it evaluates -0.60244329 <= -0.60244329 as False making this one observation have a different prediction for its scaled counterpart. All other predictions are the same.

#### Expected Results

The scaled predictions should match the original. I believe this issue is caused by how the decision path is evaluated. When I call `dtr.tree_.decision_path(X_scaled)` an error is raised informing me that I need to use a float32 and not a float64. This lower precision float might be causing the decision to be evaluated incorrectly. 

#### Versions

System:
    python: 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 14:38:56)  [Clang 4.0.1 (tags/RELEASE_401/final)]
executable: /Users/Ted/miniconda3/envs/minimal_ds/bin/python
   machine: Darwin-18.7.0-x86_64-i386-64bit

Python deps:
       pip: 19.3.1
setuptools: 41.6.0.post20191101
   sklearn: 0.21.3
     numpy: 1.17.3
     scipy: 1.3.1
    Cython: 0.29.14
    pandas: 0.25.3

user: tdpetrou
issue_15603: state: open
title: [RFC] GridSearchCV on OneClassSVM
body: It seems that this is pretty difficult to make some grid-search on `OneClassSVM`

I wanted to search the `C` parameter with a `RandomizedSearchCV` and give the `balanced_accuracy_score`. However, `OneClassSVM` will predict `1` and `-1` which will be different from the values in `y`.

Shall we have a specific way to encode the target for novelty and outlier detection algorithms?

ping @albertcthomas 
user: glemaitre
issue_15588: state: open
title: CountVectorizer integration with ColumnTransformer is unintuitive
body: Here is an issue that I came across with Count Vectorizer and its use with Column Transformer and Pipelines

https://stackoverflow.com/questions/54541490/sklearn-text-and-numeric-features-with-columntransformer-has-value-error/57970935#57970935

I found a work around using by removing the column name to be processed directly instead of just a list. 

This may be the way that it is intended to function, but if not, a simple bug fix would be just to add a simple line of code to change the list of column names to the content of the list if the length  of the list is one. 

<!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->


<!-- Thanks for contributing! -->

user: beachkrp
issue_15585: state: open
title: Should fetch_openml's categoricals be ordered?
body: We recently added an option in fetch_openml to return a DataFrame where nominals from openml would be encoded as pandas categoricals. However these categoricals are constructed as unordered. In arff, used by openml, there is no way to specify a nominal attribute without giving its values an order, so it's possible that the nominal feature should be ordered. Should we be setting the ordered flag when loading into pandas, since this is the more strict option? Ping @janvanrijn 
user: jnothman
issue_15583: state: open
title: MRG Logistic regression preconditioning
body: Partially addresses #15556.

Right now only for l-bfgs.
I added a parameter for people to confirm that it behaves as expected and easier debugging, but I think we should not add a parameter and always do this.

Might be a bit late but honestly I would like to avoid users having another 6 month of convergence warnings whenever they fit logistic regression. Also the results are much more stable / better and faster.

To demonstrate the effect:
```python
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.linear_model._logistic import _logistic_loss, _logistic_loss_and_grad
import numpy as np

X, y = make_classification(n_samples=100, n_features=60, random_state=0)
X[:, 1] += 10000
X[:, 0] *= 10000

lr = LogisticRegression(tol=1e-8).fit(X, y) # convergence warning
lr_pre = LogisticRegression(tol=1e-8, precondition=True).fit(X, y) # no warning
lr1000 = LogisticRegression(max_iter=1000, tol=1e-8).fit(X, y) # no warning, still worse solution!

print(_logistic_loss(np.hstack([lr.coef_.ravel(), lr.intercept_]), X, 2 * y - 1, 1))
# 14.289
print(_logistic_loss(np.hstack([lr1000.coef_.ravel(), lr1000.intercept_]), X, 2 * y - 1, 1))
# 13.478
print(_logistic_loss(np.hstack([lr_pre.coef_.ravel(), lr_pre.intercept_]), X, 2 * y - 1, 1))
# 12.354
```
in other words: it converges faster and to a better solution.

Multinomial
```python
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
import scipy
from sklearn.linear_model._logistic import _multinomial_loss, _multinomial_loss_grad
import numpy as np
from sklearn.preprocessing import label_binarize

X, y = make_classification(n_samples=100, n_features=60, n_classes=3, random_state=0, n_informative=6)
sample_weight = np.ones(X.shape[0])
Y = label_binarize(y, [0, 1, 2])
X[:, 1] += 10000
X[:, 0] *= 10000

lr = LogisticRegression(max_iter=100).fit(X, y) # convergence warning
lr10000 = LogisticRegression(max_iter=10000).fit(X, y) # no convergence warning, still worse. 1000 is not enough.
lr_pre = LogisticRegression(precondition=True, max_iter=100).fit(X, y)
# convergence warning, though max_iter=200 would fix it

loss, p, w = _multinomial_loss(np.hstack([lr.coef_, lr.intercept_.reshape(-1, 1)]), X, Y, 1, sample_weight=sample_weight)
print(loss) # 31.075

loss_10000, p, w = _multinomial_loss(np.hstack([lr10000.coef_, lr10000.intercept_.reshape(-1, 1)]), X, Y, 1, sample_weight=sample_weight)
print(loss_10000)  # 18.972

loss_pre, p, w = _multinomial_loss(np.hstack([lr_pre.coef_, lr_pre.intercept_.reshape(-1, 1)]), X, Y, 1, sample_weight=sample_weight)
print(loss_pre) # 18.308
```
For multinomial, it still warns with the defaults, but produces a better result than master with 10000 iterations (in which case master doesn't warn).
So this doesn't remove all warnings, but makes the solutions much more stable, and allows solving by increasing ``max_iter`` within a reasonable range (in this example).

Changing to ``n_samples=10000`` the above script produces
```
max_iter=100: 7999.571 & warn
max_iter=10000: 7999.5717 & doesn't warn
max_iter=100, precondition=True: 7734.327 & warn
max_iter=200, precondition=True: 7733.976 & doesn't warn
```

user: amueller
labels: Needs Decision: Requires decision
issue_15556: state: open
title: Scaling issues in l-bfgs for LogisticRegression
body: So it looks like l-bfgs is very sensitive to scaling of the data, which can lead to convergence issues.
I feel like we might be able to fix this by changing the framing of the optimization?

example:
```py
from sklearn.datasets import fetch_openml
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import scale

data = fetch_openml(data_id=1590, as_frame=True)
cross_val_score(LogisticRegression(), pd.get_dummies(data.data), data.target)
```
this gives convergence warnings, after scaling it doesn't. I have seen this in many places. While people *should* scale I think warning about number of iterations is not a good thing to show to the user. If we can fix this, I think we should.

Using the bank campaign data I got coefficients that were quite different if I increased the number of iterations (I got convergence warnings with the default of 100). If I scaled the data, that issue went away.
user: amueller
issue_15553: state: open
title: FYI/RFC: MLR Pipeline infrastructure and fit.transform vs fit_transform
body: I recently talked with the authors of the MLR package for R. The just changed their pipeline infrastructure to something quite similar to ours - though they developed it independently.
See this for details: https://mlr3pipelines.mlr-org.com

I wanted to point out one particular aspect of their design.
We have been having issues with fit().transform() vs fit_transform, for example in stacking and resampling.
They completely avoid the issue by having ``fit`` produce a representation of the dataset. You could say they don't have ``fit``, they only have ``fit_transform``, though it's just called ``fit`` basically.
That makes it very obvious that there are two separate transformation implemented by each model: the one on the training set, and the one on the test set, and no confusion can arise.
The method names are very different, so there is no expectations that the two would ever produce the same result.

I'm not sure this is a route we want to consider, but it would remove a lot of pain points we're currently seeing, and it seems like a cleaner design to me.

If we wanted to do something like that in sklearn we either need to stop returning ``self`` in ``fit`` which might be too much of a break in the API, even for a 1.0 release.
The other option would be to come up with another verb, and have ``fit_verb``, which we kind of have for ``fit_resample``. Though in principal it could be ``fit_transmogrify`` and do arbitrary things (like stacking).

I found it quite fascinating that months of our discussions would just resolve if we hadn't decided to have ``fit`` return ``self``...
user: amueller
issue_15551: state: open
title: Supervised discretization
body: Is there any interest to implement some supervised discretization algorithms? I would imagine that it might work better than quantile on kmeans discretization in `KBinsDiscretizer`, particularly on somewhat imbalanced classes.

There is a detailed review of supervised discretization techniques in ["A survey of discretization techniques: Taxonomy and empirical analysis in supervised learning"](https://sci2s.ugr.es/sites/default/files/ficherosPublicaciones/1469_2013-Garcia-IEEETKDE.pdf), *Garcia et al 2012*. I have only skimmed though it, it is quite extensive. Here is a figure comparing the classification accuracy using naive bayes classifier, averaged over 40 datasets , as far as I understood,

 
![1469_2013-Garcia-IEEETKDE](https://user-images.githubusercontent.com/630936/68303145-8ccaed80-00a3-11ea-9108-49481d6a6681.png)

There some of the supervised algorithms in the top left corner, have a comparable classification to "equal width" or "equal frequency" approaches but with 2-4x fewer bins.

I haven't done a proper review, but among well established methods there are for instance,
  - CAIM discretization algorithm, *Kurgan 2004* (511 citations), Python implementation available [here](https://github.com/airysen/caimcaim) but it doesn't seem much used.
  - MDLP  was proposed in https://github.com/scikit-learn/scikit-learn/pull/4801 but moved to a separate project due to implementation complexity.

Thoughts? It could also be something for scikit-learn-extra..
user: rth
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_15546: state: open
title: Use Bit Vectors for binary Vectorizers instead of float
body: For vectorizers that support `binary=True` it seems excessive to use a `float` when a single bit is enough. Switching to a Bit Vector data type could increase memory efficiency by 32x compared to a `float32`, greatly improving scalability for NLP. Such a structure might not even need to be sparse. It would probably require moving away from a Numpy CSR for this case, but perhaps it's a big enough benefit worth considering?
user: stevennic
issue_15531: state: open
title: Enforce positive sample_weight
body: As discussed in https://github.com/scikit-learn/scikit-learn/issues/12464#issuecomment-433815773 and https://github.com/scikit-learn/scikit-learn/issues/15358#issuecomment-549048650 in some specific use-cases negative `sample_weight` may be meaningful, however in most cases they should never happen.

So we may want to
 - add `force_positive=None` parameter to `_check_sample_weights`.
 - add a `assume_positive_sample_weights=True` config parameter to `sklearn.set_config` / `get_config`.

By default, `force_positive=None` would error on negative sample weight, but this check could be disabled globally with `sklearn.set_config(assume_positive_sample_weights=False)`.

With `_check_sample_weights(.., force_positive=True)` the check would always be done irrespective of the config parameter.

If there are no objections to this, please tag this issue as "Help Wanted".
user: rth
issue_15526: state: open
title: ENH: implement __sizeof__ for estimators
body: #### Reference Issues/PRs
Closes #8642 

#### What does this implement/fix? Explain your changes.
It implements `__sizeof__` for Estimators as mentioned in https://github.com/scikit-learn/scikit-learn/issues/8642#issue-216883385.

TODO:

- [x] implement and test `__sizeof__`
- [ ] implement `__sizeof__` for trees
user: stsievert
issue_15504: state: open
title: [MRG] BUG: enforce row-wise arg max of decision function=predicted class
body: 
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #14124 

#### What does this implement/fix? Explain your changes.
The estimators are traversed in reversed order to be consistent with the value returned on arg-max of decision function.

#### Any other comments?
The reproducer in the original issue, now runs successfully.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: lakrish
issue_15464: state: open
title: Failure of doctest for sklearn.cluster._bicluster.SpectralCoclustering on osx.
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
DocTest failure on osx when testing master.

```pytb
______________________________________________________________________________
___________ [doctest] sklearn.cluster._bicluster.SpectralCoclustering 
______________________________________________________________________________
____________
253         The bicluster label of each column.
254 
255     Examples
256     --------
257     >>> from sklearn.cluster import SpectralCoclustering
258     >>> import numpy as np
259     >>> X = np.array([[1, 1], [2, 1], [1, 0],
260     ...               [4, 7], [3, 5], [3, 6]])
261     >>> clustering = SpectralCoclustering(n_clusters=2, random_state=0).fit(X)
262     >>> clustering.row_labels_
Expected:
    array([0, 1, 1, 0, 0, 0], dtype=int32)
Got:
    array([0, 0, 1, 0, 0, 0], dtype=int32)
```
#### Steps/Code to Reproduce
pytest sklearn
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
System:
    python: 3.7.5 (default, Oct 25 2019, 10:52:18)  [Clang 4.0.1 (tags/RELEASE_401/final)]
executable: /anaconda3/envs/sklearndev/bin/python
   machine: Darwin-18.7.0-x86_64-i386-64bit

Python deps:
       pip: 19.3.1
setuptools: 41.6.0.post20191030
   sklearn: 0.22.dev0
     numpy: 1.16.2
     scipy: 1.3.1
    Cython: 0.29.13
    pandas: None
matplotlib: 3.1.1
    joblib: 0.14.0

<!-- Thanks for contributing! -->

user: lakrish
labels: module:test-suite: everything related to our tests
issue_15403: state: open
title: ENH fast path for binary confusion matrix
body: A patch for @GregoryMorse to benchmark.

Fixes #15388.
user: jnothman
issue_15389: state: open
title: Negative Log Loss metric fails in LogisticRegressionCV when class is unobserved in CV fold
body: #### Description
When LogisticRegressionCV is called on a multi-class dataset with a "low-data" class, some CV folds are forced to drop the rare-class entirely.  This propagates through to the `neg-log-loss` metric as an exception:

#### Steps/Code to Reproduce

```python
"""
Modify the digits dataset to be very low in 9s, to trigger
an exception in the negative log loss metric.
"""

import sklearn.datasets
import sklearn.linear_model
import numpy as np

X, y = sklearn.datasets.load_digits(return_X_y=True)

n9 = 2

y9 = y[y == 9]
X9 = X[y == 9]

X = X[y != 9]
y = y[y != 9]

X = np.vstack([X, X9[0:n9]])
y = np.hstack([y, y9[0:n9]])

multi_class = "multinomial"
scoring = "neg_log_loss"
random_state = 1

model = sklearn.linear_model.LogisticRegressionCV(multi_class=multi_class, scoring=scoring, random_state=random_state)
model.fit(X, y)

```

#### Expected Results
I would expect this to run without an exception, or raise an exception before model training is attempted.   In particular, if you assume that 0 log(0) -> 0, this should "just work" IMHO.  

#### Actual Results

```
....env/lib/python3.6/site-packages/sklearn/linear_model/logistic.py in _log_reg_scoring_path(X, y, train, test, pos_class, Cs, scoring, fit_intercept, max_iter, tol, class_weight, verbose, solver, penalty, dual, intercept_scaling, multi_class, random_state, max_squared_sum, sample_weight, l1_ratio)
   1195             scores.append(log_reg.score(X_test, y_test))
   1196         else:
-> 1197             scores.append(scoring(log_reg, X_test, y_test))
   1198 
   1199     return coefs, Cs, np.array(scores), n_iter

....env/lib/python3.6/site-packages/sklearn/metrics/scorer.py in __call__(self, clf, X, y, sample_weight)
    138                                                  **self._kwargs)
    139         else:
--> 140             return self._sign * self._score_func(y, y_pred, **self._kwargs)
    141 
    142     def _factory_args(self):

....env/lib/python3.6/site-packages/sklearn/metrics/classification.py in log_loss(y_true, y_pred, eps, normalize, sample_weight, labels)
   2164                              "y_true: {2}".format(transformed_labels.shape[1],
   2165                                                   y_pred.shape[1],
-> 2166                                                   lb.classes_))
   2167         else:
   2168             raise ValueError('The number of classes in labels is different '

ValueError: y_true and y_pred contain different number of classes 9, 10. Please provide the true labels explicitly through the labels argument. Classes found in y_true: [0 1 2 3 4 5 6 7 8]
```

#### Versions

```
System:
    python: 3.6.7 | packaged by conda-forge | (default, Jul  2 2019, 02:07:37)  [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
executable: miniconda3/envs/eda/bin/python
   machine: Darwin-18.7.0-x86_64-i386-64bit

Python deps:
       pip: 19.3.1
setuptools: 41.4.0
   sklearn: 0.21.3
     numpy: 1.17.2
     scipy: 1.3.1
    Cython: None
    pandas: 0.25.2
```

One workaround that I've been able to implement is using `functools.partial` to create a custom loss function that takes the output of `predict_proba()` and expands it by filling zeros into the missing label.  This approach assumes that the missing label is always at the end of the array, which I think is reasonable but needs to be tested.  
user: kyleabeauchamp
issue_15388: state: open
title: metrics.confusion_matrix far too slow for Boolean cases
body: #### Description
When using metrics.confusion_matrix with np.bool_ cases (e.g. with only True/False values), it is far more fast to not use list comprehensions as the current code does.  numpy has sum and Boolean logic functions to deal with this very efficiently to scale far better.
Code found in: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/_classification.py which uses list comprehensions and never checks for np.bool_ types trying to avoid all the excessive code.  This is a very common and reasonable use case in practice.  Assuming normalization and no sample weights though those also can be efficiently dealt with.

#### Steps/Code to Reproduce
```
import numpy as np

N = 4096
p = 0.5
a = np.random.choice(a=[False, True], size=(N), p=[p, 1-p])
b = np.random.choice(a=[False, True], size=(N), p=[p, 1-p])
from sklearn.metrics import confusion_matrix
for i in range(1024): confusion_matrix(a, b)
```
#### Expected Results
Fast execution time.
e.g. substituting confusion_matrix with this conf_mat (not efficient as possible but easier to read and more efficient than current library even with 4 sums, 4 logical ANDs and 4 logical NOTs:
```
        def conf_mat(x, y):
            return np.array([[np.sum(~x & ~y), np.sum(~x & y)], #true negatives, false positives
                             [np.sum(x & ~y), np.sum(x & y)]]) #false negatives, true positives
```

Or even faster with 1 logical AND with 3 sums:
```
        def conf_mat(x, y):
            truepos, totalpos, totaltrue = np.sum(x & y), np.sum(y), np.sum(x)
            totalfalse = len(x) - totaltrue
            return np.array([[totalfalse - falseneg, totalpos - truepos], #true negatives, false positives
                             [totaltrue - truepos, truepos]]) #false negatives, true positives

```

#### Actual Results
Slow execution time around 60 times slower than efficient code.  The np.bool_ case could be identified and efficient code applied otherwise in serious cases of scale, the current code is too slow to be practically usable.

#### Versions
All - including 0.21
user: GregoryMorse
issue_15381: state: open
title: Make sure the CI fails on Deprecation and Future warnings
body: Sometimes the CI passes eventhough these warnings are raised in our tests. We should make sure they fail and the issues are caught.
user: adrinjalali
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_15371: state: open
title: Strict identity parameter check in sklearn.base.clone
body: I'm trying to add sklearn-compatibility to Cython-based estimators. Running clone() on these requires a custom get_params(), that copies values to new c-objects instead of copying references.

clone() validation doesn't accept this, since it's using reference comparison "is" instead of value comparison "==" to check for equality. Shouldn't this validation be checking for values instead of references, since the clones should be permitted to have their own objects?

https://github.com/scikit-learn/scikit-learn/blob/67ea25df1f199a279dd68f91a322e951b59b35f4/sklearn/base.py#L70

In my case I add this debug print the validation loop:
```
print(name, param1, param2, type(param1), type(param2), param1 is param2, param1==param2)
```
and get values:
```
alpha 0.1 0.1 <class 'float'> <class 'float'> False True
```
Here the float reference comparison "is" fails, whereas the value comparison "==" works. Fixing the validation loop check to just "if param1 != param2:" works in my case, but could need to be modified to handle other cases.
user: anttttti
issue_15336: state: open
title: Add Sparse Matrix Support For HistGradientBoostingClassifier
body: ### Description
Hi!

I'm receiving the error below when attempting to pass a sparse matrix to `HistGradientBoostingClassifier`. The matrix is the result of using `CountVectorizer` and `TfidfTransformer` on input text.

In my case, the size of the text prohibits converting the sparse matrix to a dense one (I run out of memory).

#### Steps/Code to Reproduce

```
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.ensemble import HistGradientBoostingClassifier

df = pd.read_csv(...)

vectorizer = CountVectorizer()
tfidf = TfidfTransformer()
clf = HistGradientBoostingClassifier()

vecs = vectorizer.fit_transform(df.loc[:, "very_large_text"])
vecs = tfidf.fit_transform(vecs)

clf.fit(vecs, df.loc[:, "label"])
```

#### Expected Results
No error is thrown.

#### Actual Results
`TypeError: A sparse matrix was passed, but dense data is required. Use
X.toarray() to convert to a dense numpy array.`

#### Versions

System:
    python: 3.7.3 (default, Oct  1 2019, 18:28:53)  [GCC 5.4.0 20160609]
executable: /local_disk0/pythonVirtualEnvDirs/virtualEnv-3631eab5-084b-4139-952e-5aff594ac1bb/bin/python
   machine: Linux-4.15.0-1050-azure-x86_64-with-debian-stretch-sid

Python deps:
       pip: 19.0.3
setuptools: 40.8.0
   sklearn: 0.21.3
     numpy: 1.16.2
     scipy: 1.2.1
    Cython: 0.29.6
    pandas: 0.24.2


user: jmwoloso
labels: Hard: Hard level of difficulty
issue_15287: state: open
title: [AgglomerativeClustering] should be lax about precomputed distances
body: #### Description

Precomputed distances are treated as "unsupported" by AgglomerativeClustering, regardless of the provenance. (e.g. sklearn.metrics.euclidean_distances)

This restriction makes it impossible to pass a off-memory (e.g. memory mapped) precomputed distance matrix, even if it is valid Euclidean distance.

#### Steps/Code to Reproduce

```
import numpy as np

from sklearn.metrics import euclidean_distances
from sklearn.cluster import AgglomerativeClustering

X = np.array([[1, 2], [1, 4], [1, 0],[4, 2], [4, 4], [4, 0]])
c = AgglomerativeClustering(affinity='precomputed')
c.fit(euclidean_distances(X))
```

#### Expected Results

This probably a bit complicated, as there is no reliable way to determine if the distance matrix actually contains Euclidean distances or is complete nonsense. As precomputed matrices are a somewhat advanced feature, simply relaxing the requirements and assuming users won't pass in invalid distances might be good enough, but that's a decision to be made by the maintainers.

If the matrix was generated from sklearn.metrics, it might be possible to mark a provenance hint somewhere in the object, while it does feel a bit hacky.

#### Actual Results

```
In [21]: c.fit(euclidean_distances(X))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-21-38e0d7bfcae1> in <module>
----> 1 c.fit(euclidean_distances(X))

~/anaconda3/lib/python3.7/site-packages/sklearn/cluster/hierarchical.py in fit(self, X, y)
    823             raise ValueError("%s was provided as affinity. Ward can only "
    824                              "work with euclidean distances." %
--> 825                              (self.affinity, ))
    826
    827         if self.linkage not in _TREE_BUILDERS:

ValueError: precomputed was provided as affinity. Ward can only work with euclidean distances.
```

#### Versions

System:
    python: 3.7.3 (default, Mar 27 2019, 22:11:17)  [GCC 7.3.0]
executable: /home/user/anaconda3/bin/python
   machine: Linux-5.0.0-31-generic-x86_64-with-debian-buster-sid

Python deps:
       pip: 19.1.1
setuptools: 41.4.0
   sklearn: 0.21.3
     numpy: 1.16.2
     scipy: 1.2.1
    Cython: 0.29.12
    pandas: 0.24.2

This affects all platforms.
user: cynthia
issue_15272: state: open
title: sklearn MDS vs skbio PCoA
body: [Multi Dimensional Scaling](https://en.wikipedia.org/wiki/Multidimensional_scaling) (MDS) and Principal Coordinate Analysis (PCoA) are two names for the same dimension reduction technique*.

In [scikit-learn, MDS](https://github.com/biocore/scikit-bio/blob/master/skbio/stats/ordination/_principal_coordinate_analysis.py) is implemented with the SMACOF algorithm while in other Python libraries (such as [scikit-bio](https://github.com/biocore/scikit-bio/blob/master/skbio/stats/ordination/_principal_coordinate_analysis.py)) and most R packages offering it, it is implemented using singular value decomposition.

This is usually quite confusing for people who try out the sklearn implementation of MDS when comparing it with other MDS implementations (sklearn stands out).

How could one add another implementation of MDS in sklearn ? Or maybe create a new PCoA method ? 

cc @adrinjalali 

- [Issue #579 being discussed in the scikit-bio repository](https://github.com/biocore/scikit-bio/issues/579)
- [StackOverflow thread](https://stackoverflow.com/questions/38905833/why-is-sklearn-manifold-mds-random-when-skbios-pcoa-is-not)
- [*MDS vs PCoA](https://stats.stackexchange.com/questions/14002/whats-the-difference-between-principal-component-analysis-and-multidimensional/14017#14017)
user: maxibor
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_15263: state: open
title: Interaction terms between categorical and numerical features
body: #### Description
While it is possible to create feature interactions with the same individual preprocessing in `ColumnTransformer` via `PolynomialFeatures`, I find no (convincing) solution for interactions of features with different individual preprocessing, e.g. categorical column with a continuous numerical column.
Such interactions might improve models from `sklearn.linear_model`.

#### Code Example
```python
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, PolynomialFeatures


df = pd.DataFrame({'a': ['red', 'red', 'blue', 'blue'],
                   'b': ['high', 'low', 'high', 'low'],
                   'x': [1, 1, 1, 2],
                   'y': [2, 3, 4, 2]
                  })

# interactions for features with no individual preprocessing works fine,
# i.e. numerical ones
column_trans = ColumnTransformer(
    [('xy_num',
      PolynomialFeatures(degree=2, interaction_only=True, include_bias=False),
      ['x', 'y'])],
     remainder='drop')
column_trans.fit_transform(df)

# interactions for ohe encoded also works with helper function
cat_cat = make_pipeline(
    OneHotEncoder(),
    PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
)
column_trans = ColumnTransformer(
    [('ab_cat', cat_cat, ['a', 'b'])],
     remainder='drop')
column_trans.fit_transform(df)
```
#### Expected Results
```python
# no clue for interactions between one-hot-encoded 'a' and 'x' 
column_trans = ColumnTransformer(
    [('a_x',
      magic_pipeline(OneHotEncoder(), 'passthrough'),
      ['a', 'x'])],
     remainder='drop')
```

#### Versions
sklearn version 0.21
user: lorentzenchr
issue_15256: state: open
title: Future request: NaN instead of zeros when cossine_similairty is computed on a zero vector
body: #### Description
When you have a dataframe with a zero vector and you apply `cosine_similarity` to that, the zero-vector will give 0 similarity, when it should be NaN. I think there is a big difference getting a zero-similarity or a NaN

#### Versions
sklearn: 0.21.3
Platform: Linux-4.15.0-52-generic-x86_64-with-debian-9.9
Python: 3.7.3 
NumPy: 1.17.2
Scipy:1.3.1

user: Jakobhenningjensen
issue_15250: state: open
title: Better Documentation for Density Estimation 
body: Sort of like #14228 

scikit-learn/doc/modules/density.rst

Would be great to have references and some exposition/examples of `feature engineering` , `data modelling` and `unsupervised learning` and their relationship to density estimation. 


user: project-delphi
issue_15247: state: open
title: Should sklearn include the Equal Error Rate metric?
body: Hi there,

I have recently looked into using sklearn for biometric security models. In this field one of the most common metrics is equal error rate (EER). Surprisingly it's not out of the box in sklearn despite the long list of scores and metrics shipped with https://scikit-learn.org/stable/modules/classes.html

I have coded a piece of code to compute the EER in different contexts, such as optimizing hyperparameters and model space to minimize it. Is it worth cleaning it and including it in a pull request?
user: vnherdeiro
issue_15176: state: open
title: [WIP] Implement Gini coefficient for model selection with positive regression GLMs
body: Fixes #10003. This PR builds on top of #14300.

I plan to rebase the last 2 commits on top of #14300 from time to time.

TODO:

- [ ] Add tests
- [ ] Add documentation
- [ ] Add a helper plot_lorenz_curve function?
user: ogrisel
issue_15169: state: open
title: Avoid calculating feature importance for multiple times in SelectFromModel
body: Currently, in SelectFromModel, we calculate feature importance during ``transform``. If users do feature selection based on the training set and transform both the training set and the testing set, they'll need to calculate feature importance for multiple times. Actually calculating feature importance is sometimes time-consuming (e.g., a large xgboost model), so I think we should figure out a way to avoid this.
I don't have a solution. We can't calculate and store feature importance during ``fit`` because when ``prefit=True``, we allow users to call ``transform`` directly. We can't store feature importance during ``transform`` because we can't add/modify attributes during transform.
user: qinhanmin2014
labels: Needs Decision: Requires decision
issue_15161: state: open
title: Do we want to keep the Passive Agressive estimators?
body: We support the PassiveAggressive [classifier](https://scikit-learn.org/dev/modules/generated/sklearn.linear_model.PassiveAggressiveClassifier.html#sklearn.linear_model.PassiveAggressiveClassifier) and regressor, which simply are BaseSGD with hinge loss and a specific learning rate schedule (+ some regularization).

**I'm wondering whether we actually have a compelling reason to keep them around?** Are these estimators useful to anyone?

They are [quite poorly documented](https://scikit-learn.org/dev/modules/linear_model.html#passive-aggressive), and the only examples they're used in are just general benchmark comparison examples.

I feel like if we can't give users an incentive to use an estimator (i.e. in what cases the estimator should perform well), then we have no reason to support it. The UG doesn't do that right now.

Also note that they are apparently sub-optimal https://github.com/scikit-learn/scikit-learn/issues/1327

----

I have no strong opinion, I'm mainly genuinely curious.

We could also just move them to e.g. scikit-learn-extras. But that would be a superficial removal since we would still need to support their specific learning rate schedule as deep as in `_fast_sgd()`.

If we decide to keep them around I'll make a PR to document them because this is really bad right now.

ping maybe @ogrisel and @GaelVaroquaux since you have reviewed the original PR  https://github.com/scikit-learn/scikit-learn/pull/1259
user: NicolasHug
issue_15154: state: open
title: Transform target when using SelectFromModel
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
I am using `SelectFromModel` and `LassoCV` for feature selection. `LassoCV` uses cross-validation, so it calculates the mean square error for the test set on each fold. Now, since my final model uses y-transformation, I'd like to be able to do the same when selecting features. How can I do that?

The example code shows that it is not possible to pass a `TransformedTargetRegressor` to `SelectFromModel`. I have also shown that fitting on y vs. log(y) produces different results. 

#### Steps/Code to Reproduce
```python
from sklearn.datasets import make_gaussian_quantiles
from sklearn.feature_selection import SelectFromModel
from sklearn import linear_model
from sklearn import metrics
from sklearn.preprocessing import StandardScaler
from sklearn.compose import TransformedTargetRegressor

X, y = make_gaussian_quantiles(random_state=0, n_features=10, n_samples=100)
st = StandardScaler()
X = st.fit_transform(X)
y= y*3+5
clf = linear_model.LassoCV(cv=5)

trans_clf = TransformedTargetRegressor(regressor=linear_model.LassoCV(cv=5),
                                       func=np.log,
                                       inverse_func=np.exp)

#sfm = SelectFromModel(trans_clf, threshold=0.01)  # gives error
sfm = SelectFromModel(clf, threshold=0.01)
sfm.fit(X, y)             # selects 5 features
#sfm.fit(X, np.log(y))    # selects 2 features

n_features = 10

while n_features > 5:
    sfm.threshold += 0.01
    X_transform = sfm.transform(X)
    n_features = X_transform.shape[1]

print("selected features: ", n_features)
```
#### Expected Results
I'd like to be able to pass a `TransformedTargetRegressor` to `SelectFromModel`

#### Actual Results

1. If I pass the TransformedTargetRegressor, I get the error: 
_ValueError: The underlying estimator TransformedTargetRegressor has no `coef_` or `feature_importances_` attribute. Either pass a fitted estimator to SelectFromModel or call fit before calling transform._

2. If I transform the target variable when fitting the `sfm`, the features will be selected for a `log(y)~X` regression, which is not what I want. I'd like to get features for the `y~X` regression when y is transformed.

#### Versions
python: 3.7.3
sklearn: 0.21.2
user: apptimise
issue_15149: state: open
title: Parameter searches: folds used in evaluate_candidates aren't consistent across calls
body: If I use a custom CV iterator with `suffle=True, random_state=None`, then different folds will be generated for each call to `evaluate_candidates()`.

This isn't an issue for GridSearchCV or RandomizedSearchCV since these only call `evaluate_candidates()` once. But this is fundamentally wrong for e.g. Successive Halving #13900, which repeatedly call `evaluate_candidates` but assumes the folds to be always the same (if `resource != n_samples`). This is even more of an issue if we implement warm start for SH: estimators would not be warmstarted on the *same data*.

ping in particular @jnothman , I think this is something you noted before (but can't remember where, sorry)?
user: NicolasHug
issue_15136: state: open
title: [WIP]  Refactoring logic for chosing eigen solver in spectral clustering (#14713)
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
Enhancement #14713
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
Trying to simplify the logic for selecting a solver in spectral_clustering.
The scheme for deciding which solver to use remains the same. The refactoring goal is not to change the algorithm for chosing the solver. Rather, it tries to make the code more clear.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: alexshacked
issue_15125: state: open
title: Notes on warmstarting GridSearchCV and SuccessiveHalving
body: Throwing some thoughts on warm starting, grid search and successive halving (#13900 ). In particular: warm-starting in grid search is very different from warm-starting in SH. Details below.

I think this could be of interest to @jnothman , @amueller, @adrinjalali  and @ogrisel .

# Warm start in GridSearchCV

GridSearchCV currently does not support warm-starting, which is a shame since it wastes resources. Supporting it would allow to get rid of the EstimatorCV objects.


Consider the folloging param_grid, where b is a warmstartable parameter:

```py
param_grid = {
    'a': [1, 2],
    'b': [3, 4]
}
```

The trick proposed by @jnothman in https://github.com/scikit-learn/scikit-learn/pull/8230 is to transform the list generated by `ParameterGrid` from

```py
[{'a': 1, 'b': 3}, {'a': 1, 'b': 4}, {'a': 2, 'b': 3}, {'a': 2, 'b': 4}]
```

to

```py
[[{'a': 1, 'b': 3}, {'a': 1, 'b': 4}],
 [{'a': 2, 'b': 3}, {'a': 2, 'b': 4}]]
```

This way, in `evaluate_candidates()`, instead of cloning the estimator 4 times (once per dict), we only clone it twice (once for per sublist, where warm_start can be leveraged).

The transformation isn't necessarily obvious, especially considering that the values of 'a' aren't hashable in general. 

From a (private) API point of view, there are many nasty ways of doing it, I couldn't come up with a clean version so far.

# Warm start in RandomizedSearchCV

`RandomizedSearchCV` simply cannot support warm starting. It doesn't make sense in this case since the non-warmstartable parameters are sampled at random, and there is no way to construct groups of parameters that can be warmstarted together.

# Warm start in SuccessiveHalving

Warm start in SH is possible (for both the grid and random version), but it is completely different from the way we can warm-start a `GridSearchCV`.

Let's say we budget on the number of trees of a GBDT and `n_trees` is a warmstartable parameter (I'm using `n_trees` instead of `max_iter` to avoid confusion with the iterations of the SH process).

Warm-starting here consists of re-using at SH iteration `i + 1` the estimators that were ran at SH iteration `i`, since the only parameter that differs between them is `n_trees`: 

![IMG_20191003_173112  | height=10](https://user-images.githubusercontent.com/1190450/66166352-14d15980-e605-11e9-8a55-e27f5342c4a5.jpg)

The dashed lines represent the re-use of an estimator from one SH iteration to the next.

I hope it's clear that this kind of warm-starting is very different from the one that `GridSearchCV` can leverage.

Technically:

- `GridSearchCV` supports warm-starting for a **single call** to `evaluate_candidates(candidate_params)`. The warm-starting is done from one candidate to another.
- `SH` supports warm-starting for **multiple successive calls** to `evaluate_candidates(candidate_params)`.


That version should be relatively easy to implement in a very hacky and non-backward compatible way. But a clean version would require much more work and possibly require a whole new API to `BaseSearchCV`.

# Supporting both kinds of warm-starting

(this is a digression)

Since the nature of the warmstarting is so different, I think it should be possible to support both (i.e. both GS and SH).

However, if we ever support both, we should definitely deactivate the GS warmstart when doing SH warmstart: doing GS warm-start at a given SH iteration would mean that an estimator `x` would "become" another estimator `y` (meaning, not being cloned) because of the GS warm-start. But if that estimator `x` is one of the survivors for iteration `i + 1`, tough luck: it's lost.

In any case, that's not even an issue unless multiple parameters can be warm-started on the same estimator which is rare.

# New warm-start API???

It looks like it was decided during the sprint to support a new warm-start API for estimators, introducing a new `fit` parameter `warm_start_with` (https://github.com/scikit-learn/scikit-learn/pull/8230#issuecomment-468907013)

The transition from the old API to the new one should be pretty straightforward. I implemented a basic version for GBDTs in https://github.com/scikit-learn/scikit-learn/pull/15105

This new API allows tools like GS and SH to automatically leverage warm-starting, without needing the user to explicitly ask for it.
user: NicolasHug
issue_15123: state: open
title: A common private module for differentiable loss functions used as objective functions in estimators
body: Currently each model that needs it defines their own losses, it might be useful to put them all in one place to see if anything could be re-used in the future.

In particular, losses are defined in,
 - HistGradientBoosting: `sklearn/ensemble/_hist_gradient_boosting/loss.py`
 - GradientBoosting: `sklearn/ensemble/_gb_losses.py`
 - SGD: `sklearn/linear_model/sgd_fast.pyx`
 - GLM: https://github.com/scikit-learn/scikit-learn/pull/14300 would add `sklearn/linear_model/_glm/distributions.py` 
 - MLP: `sklearn/neural_network/_base.py:LOSS_FUNCTIONS`
 - losses implementations in `sklearn.metrics`
 - and somewhat related RandomForest: `sklearn/tree/_criterion.pyx`

This issues proposed to progressively put most of them (except for RF criterions) in a single private folder `sklearn/_loss` to see what could be made less redundant there.

In particular, for the GLM PR in https://github.com/scikit-learn/scikit-learn/pull/1430 this would allow breaking circular dependency between the `sklearn.linear_model` and `sklearn.metrics` modules https://github.com/scikit-learn/scikit-learn/pull/14300#discussion_r329370776

@NicolasHug I don't know if you already had some plans about unifying new and legacy gradient boosting losses and if this would go in the right direction. cc @ogrisel 

Weakly related to https://github.com/scikit-learn/scikit-learn/issues/5044 https://github.com/scikit-learn/scikit-learn/issues/3481 https://github.com/scikit-learn/scikit-learn/issues/5368
user: rth
issue_15118: state: open
title: [Feature suggestion] Extend VarianceThreshold with additional dispersion metrics
body: The `VarianceThreshold` class currently has limited applicability for data sets with features in different units/scales beyond acting as a zero-variance filter, due to a unified variance threshold being unlikely to work well.

I would like to suggest enhancing this class with the option of alternative dispersion metrics. More specifically, I have in mind the [coefficient of variation](https://en.wikipedia.org/wiki/Coefficient_of_variation).
Since it is dimensionless, this metric would allow the creation of a filter that conceptually targets "low variance" features in a more generally usable way, and seems straightforward to implement. The largest point of contention would be what to do to features with mean zero (or extremely small).
Naturally, the class would default to the current behavior for backward compatibility. 

Does this seem like a worthwhile addition? If so, I would like to volunteer for it :)
user: kacricon
issue_15077: state: open
title: Implement GeneralNB
body: Open an issue to discuss the implementation of GeneralNB (combine different naive bayes estimators to support different types of features)
Related issue: #10856 #12957
user: qinhanmin2014
issue_15075: state: open
title: Feature selection based on permutation importance
body: We now support permutation importance, maybe it's reasonable to support feature selection method based on it (e.g., remove features whose premutation importance < 0).
user: qinhanmin2014
issue_15050: state: open
title: [MRG] ENH Adds warning with pandas category does not match lexicon ordering
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Resolves https://github.com/scikit-learn/scikit-learn/issues/14953

#### What does this implement/fix? Explain your changes.
Gets the `dtypes` in `_fit` and checks only when it has "categories".

#### Any other comments?
The fun part begins when we try to get the encoders to respect pandas categories (when they are numerical) :)

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: thomasjpfan
labels: Needs Decision: Requires decision
issue_15032: state: open
title: [WIP] Fixing data leak with warm starting in GBDT
body: ### Reference Issues/PRs

Fixes #14034 

#### What does this implement/fix? Explain your changes.

Instead of saving a RandomState instance, which is mutated after use, an integer is saved. This ensures. A small test is added to check that the random seed is the same or different (depending on the parameters).

#### Any other comments?

There are currently 6 failing tests:
* 4 are about performance, which is worse in settings with no warm starting
* 2 are hard-coded tests

I also fixed the random seed for the random mask (out-of-bag samples, [relevant lines](https://github.com/johannfaouzi/scikit-learn/blob/5012dac2cc65b21518cbbfb7453a1150e61a0732/sklearn/ensemble/gradient_boosting.py#L1587-L1588)) (I was a bit scared at looking at the Cython function). I don't know if it is necessary or if it is a mistake.

Feedback is welcomed.

user: johannfaouzi
issue_15019: state: open
title: cross_val_score threshold adjustment
body: Apologies if this has been brought up before but a search through current and closed issues didn't return anything to indicate it had. 

I have a use case wherein I have picked an optimal decision threshold based on the intersection of sensitivity and PPV when plotted as a function of the decision threshold. The point of intersection corresponds with a decision threshold of 0.3. I would like to run cross_val_score with a LogisticRegression model that uses a threshold of 0.3 to calculate a distribution of sensitivity and FPR (separate issue, [here](https://github.com/scikit-learn/scikit-learn/issues/5516#issuecomment-532932010)). However, it doesn't seem as though the threshold can be varied in the context of cross_val_score or at the estimator level. Might be nice to have that functionality if possible. 

Thanks to all the contributors who work on this project : )

-Tommy
user: tjdurant
issue_15008: state: open
title: Segmentation fault with SVC with probability = True and degree 3
body: I am using SVC with predict proba with below hyper-parameter on large data set. it is giving segmentation fault error. code works well for smaller data set.
I don't think its memory issue we are using 96 GB RAM. I have also monitored system using htop during run & I didn't gave impression of memory issue.

**hyper-parameter** :
SVC(kernel='poly',gamma=100,C=10,degree=3,decision_function_shape='ovo',probability=True,random_state=42,verbose=True) 
**No of Features** : 768
**No of rows** : 2.1 million
user: rshah1990
issue_14997: state: open
title: QDA Classifier: divide by zero if data not of full rank 
body: #### Description
When using the QDA classifier with data that is not of full rank, matching does not work and and a divide by zero error is raised. An example dataset can be found in the code below.

#### Steps/Code to Reproduce
```python
import numpy as np
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis

np.seterr(all="warn")


def dataset_fixed_cov():
    n, dim = 300, 2
    np.random.seed(0)
    C = np.array([[0.0, -0.23], [0.83, 0.23]])
    X = np.r_[
        np.dot(np.concatenate((np.random.randn(n, 1), np.zeros((n, 1))), axis=1), C),
        np.dot(np.random.randn(n, dim), C) + np.array([1, 1]),
    ]
    y = np.hstack((np.zeros(n), np.ones(n)))
    return X, y


X, y = dataset_fixed_cov()
qda = QuadraticDiscriminantAnalysis(store_covariance=True)
y_pred = qda.fit(X, y).predict(X)

print("Correct: {}/{}".format(np.sum(y==y_pred), 600))
```

#### Expected Results
No divide by zero errors occur and the prediction works correctly.

#### Actual Results
Prediction fails for all the samples in the class and following errors are raised:

```
/usr/local/lib/python3.7/site-packages/sklearn/discriminant_analysis.py:693: UserWarning: Variables are collinear
  warnings.warn("Variables are collinear")
/usr/local/lib/python3.7/site-packages/sklearn/discriminant_analysis.py:717: RuntimeWarning: divide by zero encountered in power
  X2 = np.dot(Xm, R * (S ** (-0.5)))
/usr/local/lib/python3.7/site-packages/sklearn/discriminant_analysis.py:717: RuntimeWarning: invalid value encountered in multiply
  X2 = np.dot(Xm, R * (S ** (-0.5)))
/usr/local/lib/python3.7/site-packages/sklearn/discriminant_analysis.py:720: RuntimeWarning: divide by zero encountered in log
  u = np.asarray([np.sum(np.log(s)) for s in self.scalings_])
```

#### Versions
```
System:
    python: 3.7.4 (default, Sep 12 2019, 15:40:15)  [GCC 8.3.0]
executable: /usr/local/bin/python
   machine: Linux-5.2.14-arch2-1-ARCH-x86_64-with-debian-10.1

Python deps:
       pip: 19.2.3
setuptools: 41.2.0
   sklearn: 0.21.3
     numpy: 1.17.2
     scipy: 1.3.1
    Cython: None
    pandas: None
```
user: thielepaul
issue_14984: state: open
title: [MRG] Sorting ordering option in OrdinalEncoder
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
Fixes #14954

<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? 
Added the option `categories='lexicographic'` and added corresponding warning

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: venkyyuvy
labels: Needs Decision: Requires decision
issue_14977: state: open
title: iForest - support for height limit (hlim) evaluation parameter
body: The current fitting & predicting with iForest follows the original ["Isolation paper" paper from 2008](https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/icdm08b.pdf). Every iTree is allowed to grow to maximum of `ceil(log_2(nr_samples))` levels implemented by [setting](https://github.com/scikit-learn/scikit-learn/blob/dd6e5f8a0ce6b498f2704adb2f9989769903d462/sklearn/ensemble/iforest.py#L273) the `max_depth` parameter of the underlying `ExtraTreeRegressor` (see also #12051).

The ["Isolation-based Anomaly Detection" paper from 2012](https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/tkdd11.pdf) from the same authors amends this approach and lets iTrees grow fully but introduces another `hlim` (height limit) "evaluation parameter" applied during calculation of outlier scores with the already fitted trees. It limits the iTrees depth, but only at the level of evaluation/prediction. Authors show that by controlling this parameter one can control "granularity of anomaly scores" (what I understand as tendency of considering isolated data clusters as anomalous or not). They set `hlim` to `nr_samples-1` by default, but in comparisons often use `hlim=1`. 

Is this intentionally not implemented? If not, wouldn't it make sense to implement this flavor of fitting & predicting (e.g. by allowing setting `max_depth = None` in which case another parameter, `hlim` would be used at either its default or set value in `predict()`) ?



user: pmarko1711
issue_14966: state: open
title: Deprecate TfidfVectorizer
body: Partially addresses https://github.com/scikit-learn/scikit-learn/issues/14951

Tentatively deprecates `TfidfVectorizer` in favor of using `CountVectorizer` in a pipeline with `TfidfTransformer`. The main argument in favour of the deprecation is the complexity of the API and the number of parameters in the `TfidfVectorizer` class. That would also make the use of `CountVectorizer` and `HashingVectorizer` more symetric.

This will affect a lot of uses as `TfidfVectorizer` is frequently used. At least 3 approvals would probably be good as this is potentially controversial.

TODO:
 - [ ] make sure the documentation is fully consistent. I can do it once we are sure we want to deprecate it.
user: rth
labels: Needs Decision: Requires decision
issue_14963: state: open
title: Pass through non-ndarray object that support nep13 and nep18: vaex+sklearn out of core
body: # Intro

At EuroScipy we (@ogrisel, @adrinjalali @JovanVeljanoski and me) discussed the options of 'dataframe in (same) dataframe out' for sklearn. Also, this was a popular topic at the dataframe summit (https://datapythonista.github.io/blog/dataframe-summit-at-euroscipy.html cc @datapythonista). 

The idea was, to experiment first with some transformers, so see if we can get that to work with a vaex dataframe. The proof of concept is in this PR: https://github.com/vaexio/vaex/pull/415 

The result is that we/vaex can reuse sklearn without any modification and it works out of core (works on a billion rows for instance).

# How does it work?

## fit
By monkey patching (or using this PR), we pass through objects in check_array that are non-ndarray, but follow [NEP13](https://numpy.org/neps/nep-0013-ufunc-overrides.html) and [NEP18](https://numpy.org/neps/nep-0018-array-function-protocol.html). The idea being that if you want to behave like numpy, we will not try to convert you.

Vaex implements NEP13 and NEP18 in the referenced PR, and for instance will intercept np.sum/np.std/np.var/np.isnan etc. During for instance StandardScalar.fit, it will use vaex for the computations.

## transform
Executing StandardScalar.transform, the dataframe gets modified by building up virtual columns (we track what operations are being performed, we don't compute). The result is a dataframe with virtual columns that will be computed on the fly when needed.

# What do we get out of this?

During the fit, no memory copies are being made (at least for the examples in the vaex PR) and we get multithreading for free. The standard scalar is 5x faster out of the box and will take virtually no memory (memory-mapped data).

After the transform, we don't have just the numerical result, we have the expressions that will lead to the numerical result. These expressions can be jitted using Numba, Pythran, of by using CuPy for extra performance or GPU acceleration. Also, since we don't compute the result, we don't take up any memory.

Also, instead of having a 'nameless' ndarray, we now have a dataframe with meaningful column names.

# Conclusion / Question

Should sklearn pass through object following NEP13/NEP18? If yes, there probably needs to be a way to explicitly say 'now i need a real in memory c/fortran-contiguous array'. 

I doubt that this PR will be the final step in getting for instance vaex and sklearn or other dataframe/numpy like libraries to work together*, I hope that at least it gives food for thought. I can also imagine having an even stricter opt-int than just NEP13 and NEP18 (another magic dunder?).


*) One issue I see is in the PCA, which is using scipy.linalg.svd, which vaex cannot intercept. If numpy.linalg.svd was used, vaex could intercept that and pass it on to for instance dask. 

# Demo notebook screenshot
![image](https://user-images.githubusercontent.com/1765949/64768010-0aadc500-d549-11e9-8c9b-a501d66c5e99.png)

user: maartenbreddels
issue_14954: state: open
title: OrdinalEncoder: Deprecate automatically assuming lexicographic ordering
body: Currently, using OrdinalEncoder with a string-valued feature, and without `categories` explicitly specifying an order, means that OrdinalEncoder will number the categories according to their lexicographic ordering.

This is not appropriate if the categories have a natural ordering (e.g. ['Green', 'Amber', 'Red']) that can be harnessed by the downstream estimator.

Rather, we should allow the user to specify `categories='arbitrary'` or `categories='lexicographic'` or something to explicitly state that lexicographic ordering is okay. When the user specifies `categories='auto'` for a string-valued feature, OrdinalEncoder should raise a warning along the lines of `DeprecationWarning("From version 0.24, OrdinalEncoder's categories='auto' setting will not work with string-valued features, and categories='arbitrary' or an explicit category order will be required.")`
user: jnothman
labels: Easy: Well-defined and straightforward way to resolve
issue_14953: state: open
title: Handle pd.Categorical in encoders
body: In `sklearn.preprocessing._encoders._BaseEncoder`, columns with pd.Categorical dtype are converted to arrays.

https://github.com/scikit-learn/scikit-learn/blob/03ea20db0f9585fa0d44f4d3cae4b4c4a7c7f235/sklearn/preprocessing/_encoders.py#L60

If the `categories` ordering is explicitly specified by the user to the constructor of `OneHotEncoder` or `OrdinalEncoder`, then this is fine... but if 'auto' is used, lexicographic ordering will be assumed, disregarding the encoding order determined by the Categorical dtype.

I propose that we raise a warning if:
* a feature passed to OneHotEncoder or OrdinalEncoder has a Pandas Categorical dtype (is there a way to duck type this?)
* the categories for that feature are 'auto'
* the lexicographically sorted features do not match the feature's `dtype.categories` ordering.

The warning might be something like `UserWarning("'auto' categories is used, but the Categorical dtype provided is not consistent with the automatic lexicographic ordering")`... or else something more intelligible.

We may change this warning to a `FutureWarning` with `"From version 0.24 the category ordering specified by a Categorical dtype will be respected in encoders."`
user: jnothman
labels: Moderate: Anything that requires some knowledge of conventions and best practices
Breaking Change: Issue resolution would not be easily handled by the usual deprecation cycle.
issue_14951: state: open
title: Text vectorizers clean-up / TfidfVectorizer deprecation
body: As mentioned by @jnothman in https://github.com/scikit-learn/scikit-learn/pull/14748#issuecomment-530160881,

> I'm also happy to slowly deprecate TfidfVectorizer because it provides no
benefit over a pipeline and creates much confusion. I've seen users do
weird stuff like compare TfidfVectorizer to CountVectorizer but use
different params.

+1 on my side to deprecate it.

I would also deprecate `norm` parameter (for removal) in `HashingVectorizer` as its combination with TfidfTransformer using default parameters currently produces nonsense results due to `norm='l2'`. That would also resolve https://github.com/scikit-learn/scikit-learn/issues/6972

Generally I have also seen experienced Python users do very weird things with  text vectorizers (particularly as soon as customization is involved). Taking some time to think what how we would like its API to look ideally for 1.0 and if we can get partially there without major disruption would, I think, be useful.

In particular, I find the way we currently suggest customizing the behavior by subclassing `CountVectorizer` to update the analyzer is really awkward. I am pondering on some way to separate the pipeline that processes a single document and returns n-grams, from the `CountVectorizer` class. Basically making passing `analyzers` easier while re-using part of the existing processing.

Anyway, we can start with easy deprecations.
user: rth
labels: Needs Decision: Requires decision
issue_14947: state: open
title: feature: verbose flag in cluster.DBscan
body: would it be possible to get *verbose* "back" as a flag in DBscan. seems like it was there in earlier releases:

https://ogrisel.github.io/scikit-learn.org/sklearn-tutorial/modules/generated/sklearn.cluster.DBSCAN.html
user: sloev
issue_14942: state: open
title: Warning the user of bad default values, starting by dbscan.eps
body: Related to #13570, closes #13541.

This PR proposes adding a warning for parameters such as `dbscan.eps` which have no good default value.

It also adds a utility function which can be used potentially for other classes/parameters we'd like the user to explicitly specify a value for.

Since it's a deviation from our usual no warning with defaults policy, pinging @scikit-learn/core-devs 
user: adrinjalali
labels: Needs Decision: Requires decision
issue_14897: state: open
title: Should we make most functions private?
body: We have a lot of functions referenced in our [API reference](https://scikit-learn.org/dev/modules/classes.html).

For our own sanity, I would like to propose to make private any function that either:

- has an obvious estimator equivalent, e.g. `k_means` or `fastica`. There should be one obvious way to do things. (Some of them might be subject to debate, like `scale` which is arguably convenient.)
- isn't immediately useful for users. For example, I don't know much about OPTICS but I feel like `cluster_optics_xi` should be private (it's not even mentioned in the user guide).

There are plenty of other functions, and even more that aren't in the API ref.

----

Motivation:
`k_means` and `fastica` are making my life difficult right now as I'm working on #13603. `KMeans.fit()` does nothing but calling `k_means` which does all the work.

I cannot change **anything** to the `KMeans` class now (e.g. add an attribute),  because I would I need to change the signature / interface / behaviour of `k_means`, which is a public function.

(I think that in general the design of having the estimator call a public helper is bad (it should be the other way around), but that's another issue.)
user: NicolasHug
issue_14886: state: open
title: RFC Deprecate RidgeCV(cv=cv)?
body: I think we should deprecate ``RidgeCV`` for ``cv!=None``. It adds weird code paths and doesn't add anything over GridSearchCV. Also, changing ``cv`` radically changes the behavior in this estimator.
I think deprecating ``cv`` here would make the behavior much clearer, and probably alert some people to the fact that they are just doing standard grid-search.
user: amueller
labels: Needs Decision: Requires decision
issue_14882: state: open
title: SGDRegressor is slower than 'Ridge'
body: I have been experimenting with `SGDRegressor` estimator to show if it is faster for large training set as compared to `Ridge` and **always finding that it is slower**.

This is quite opposite to [what is recommended in the documentation for `SGDRegressor`](https://scikit-learn.org/stable/modules/sgd.html#sgd).

I have generated the same dataset using `make_regression`, forced `Ridge` to use `cholesky` solver for a linear algebra-based method, compared the time and RMSE score. I have limited the `max_iter` of `SGDRegressor` to a rather low value.

Please see the illustrative code below. Is something missing or `SGDRegressor` is not as fast as advertised?

**With `Ridge`,**

```python
from time import time
import numpy as np
from sklearn.linear_model import Ridge
from sklearn.datasets import make_regression
from sklearn.metrics import mean_squared_error


reg_ridge = Ridge(alpha=0.001,solver='cholesky')
for size in [10**i for i in range(3,7)]:
    prob=make_regression(n_samples=size,n_features=50, n_informative=45,noise=1.0,random_state=101)
    X,y=prob
    y=y.reshape(size,1)
    print("Size of the problem: ",X.shape)
    t1 = time()
    reg_ridge.fit(X,y)
    t2 = time()
    t_delta=round(1e3*(t2-t1),3)
    print(f"Took {t_delta} milliseconds")
    rmse = np.sqrt(mean_squared_error(y,reg_ridge.predict(X)))
    print("RMSE:", rmse)
    print()
```
**Returns (a typical run)**

```
Size of the problem:  (1000, 50)
Took 1.996 milliseconds
RMSE: 0.947993242263568

Size of the problem:  (10000, 50)
Took 5.934 milliseconds
RMSE: 0.9917519911562808

Size of the problem:  (100000, 50)
Took 58.843 milliseconds
RMSE: 1.0033899481015647

Size of the problem:  (1000000, 50)
Took 586.465 milliseconds
RMSE: 0.999758249692755
```

**With `SGDRegressor`,**

```python
from time import time
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import SGDRegressor
from sklearn.datasets import make_regression
from sklearn.metrics import mean_squared_error

sgd_reg = SGDRegressor(alpha=0.001,max_iter=20,tol=0.01,eta0=0.1)
scaler = StandardScaler()
for size in [10**i for i in range(3,7)]:
    prob=make_regression(n_samples=size,n_features=50, n_informative=45,noise=1.0,random_state=101)
    X,y=prob
    X = scaler.fit_transform(X)
    print("Size of the problem: ",X.shape)
    t1 = time()
    sgd_reg.fit(X,y)
    t2 = time()
    t_delta=round(1e3*(t2-t1),3)
    print(f"Took {t_delta} milliseconds")
    rmse = np.sqrt(mean_squared_error(y,sgd_reg.predict(X)))
    print("RMSE:", rmse)
    print()
```
**Returns (a typical run),**

```
Size of the problem:  (1000, 50)
Took 2.991 milliseconds
RMSE: 1.1049192408039388

Size of the problem:  (10000, 50)
Took 23.937 milliseconds
RMSE: 1.1751862949746366

Size of the problem:  (100000, 50)
Took 194.429 milliseconds
RMSE: 1.115665551451256

Size of the problem:  (1000000, 50)
Took 1875.977 milliseconds
RMSE: 1.1086122078290421
```

user: tirthajyoti
issue_14859: state: open
title: [WIP] - Precision and recall at k
body: #### Reference Issues/PRs

Example: Fixes #7343. To replace PR #4975.

Blocked by #14860, requires #14862 merge

#### What does this implement/fix? Explain your changes.
Implement "recall_at_k" and "precision_at_k" measures for multiclass/multilabel classification, accepting sparse format.

#### Any other comments?
Thx guys for your awesome work.
I initially checked this PR: https://github.com/scikit-learn/scikit-learn/pull/4975 and decided to make a new PR to adapt it to sparse representations, and benefit from features from already present 'precision_score' and 'recall_score' functions (types of averages, labels, sample_weights).

About the context in which it is useful:
- "recall_at_k" in multiclass context will give insight about the number of guess you have to make to have the right answer. It is very useful when you want to narrow down the number of classes that a person will then manually validate: with a 0.95 recall at 3, you are 95% certain of displaying the right choice to the user if you display your 3 best guesses.

- "precision_at_k" in multilabel context will give you insight of the proportion of valid predictions you make when you have a fixed number : 
Note: for samples with nb_labels < k, there will necessary have a negative impact (you can't have a good guess when there is no valid option)

As a whole these are useful metrics when you make predictions in multiclass/mutilabel context and that you have a tolerance on false-positives and want to tune how many wrong guesses you're allowing youself to make.

user: leonardbinet
issue_14829: state: open
title: OpenML: avoid warning when all versions are inactive
body: Currently when loading an OpenML dataset marked as "in preparation" we raise the following warning,
```
sklearn/datasets/openml.py:655: 
   UserWarning: Version 1 of dataset <dataset> is inactive, meaning
   that issues have been found in the dataset. Try using a newer
   version from this URL:
```

We should avoid raising this warning when all versions of a dataset are "in preparation", given that it's the case the vast majority of OpenML datasets are in that status. It doesn't mean that anything is wrong with it, only that it's hasn't been manually approved by a OpenML core dev (cf https://github.com/openml/OpenML/issues/794)

In particularly we currently get a warning due to this in https://github.com/scikit-learn/scikit-learn/pull/14300

cc @janvanrijn
user: rth
issue_14748: state: open
title: Added option to use standard idf term for TfidfTransformer and TfidfVectorizer
body: The TfidfTransformer and TfidfVectorizer use an idf that is not the standard textbook definition. I added a parameter to both as an option to use the standard definition, while leaving the previously defined idf term as the default. I created some test cases to make sure it works but I didn't include them in tests/test_text.py. Should I include them there?
user: mitchelldehaven
issue_14747: state: open
title: Tree: node_value could be float instead of double
body: #### Description

Predict function in [_tree.pyx](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/tree/_tree.pyx#L796) is comparing a float feature to a double threshold (node_value). The double threshold could be replaced by a float threshold using the following function:

```
import numpy

def float_threshold(dy):
    fy = numpy.float32(dy)
    if fy == dy:
        return fy
    if fy < dy:
        return fy
    eps = max(abs(fy), numpy.finfo(numpy.float32).eps) * 10
    nfy = numpy.nextafter([fy], [fy - eps], dtype=numpy.float32)[0]
    return nfy
```

More explanations: [Tricky detail when converting a random forest from scikit-learn into ONNX](http://www.xavierdupre.fr/app/mlprodict/helpsphinx/notebooks/onnx_float_double_skl_decision_trees.html).

#### Expected Results
Exactly the same results but with less memory consumption and maybe with some speed gain as no cast to double would be required when comparing both values.

#### Versions

Any version >= 020.

user: sdpython
issue_14725: state: open
title: Stratify subsampling when binning for histogram GBT
body: In the GBDT code, we subsample the data for finding the bin thresholds efficiently.

I think this subsampling should be stratified for classification.

(It should also depend on the sample weights once we support it).

WDYT @adrinjalali 	@ogrisel 	?
user: NicolasHug
labels: Needs Decision: Requires decision
issue_14720: state: open
title: Private instances exposed as public attributes
body: Something that came up as part of the GLM PR in https://github.com/scikit-learn/scikit-learn/pull/14300#discussion_r316683319

@rth 

> We have other examples of such private but documented behavior, existing for the same reasons:
> -    DecisionTreeClassifier.tree_ is a sklearn.tree._tree.Tree object
> -    Birch.root_ is a _CFNode
>
> meaning that we don't guarantee backward compatibility on those, but you can use them if you know what you are doing.

@NicolasHug

> I'm OK with documenting the Link class as long as the GeneralizedLinearRegressor isn't exposed, but if we ultimately expose is, I don't think we should document the use of private API, especially for init arguments.
> Regarding attributes: I would argue that we should not expose tree_ and root_ as public attributes if these are subject to change without deprecation. We claim that the public attributes should be backward-compatible.

opening this issue for further discussion.

user: rth
issue_14718: state: open
title: Changed behavior for Sparse DictVectorizer regarding Explicit Zeros in Scipy 1.3.X
body: <!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
DictVectorizer behaves differently when passed an input with explicit zeros with Scipy 1.3.X than it did with SciPy 1.2.1. 

#### Steps/Code to Reproduce
1. Create two conda environments with scikit-learn 0.21.3 but one with SciPy 1.3.1 and the other with SciPy 1.2.1  as follows: 

```bash
conda create -n sksp12 -c conda-forge scikit-learn scipy=1.2.1
conda create -n sksp13 -c conda-forge scikit-learn scipy
```

2. Run the following snippet of code in both environments:

```python
from sklearn.feature_extraction import DictVectorizer
test_feat_dict_list = [{'a': 1.0, 'b': 1.0, 'c': 0.0}, {'a': 0.0, 'b': 1.0, 'c': 1.0}]
test_dict_vectorizer = DictVectorizer()
Xtest = test_dict_vectorizer.fit_transform(test_feat_dict_list)
Xtest
```

3. The results are as follows:

SciPy 1.3.1: 
```bash
<2x3 sparse matrix of type '<class 'numpy.float64'>'
	with 6 stored elements in Compressed Sparse Row format>
```

SciPy 1.2.1: 
```bash
<2x3 sparse matrix of type '<class 'numpy.float64'>'
	with 4 stored elements in Compressed Sparse Row format>
```

Note that the SciPy 1.3.1 version has 6 stored elements rather than 4 as in the SciPy 1.2.1 version. The main reason for the change is [this PR](https://github.com/scipy/scipy/pull/7827) that _fixes an issue where fancy indexing wasn't preserving explicit zeros_. This affects [this specific line](https://github.com/scikit-learn/scikit-learn/blob/0.21.X/sklearn/feature_extraction/dict_vectorizer.py#L198) in the `DictVectorizer` code which previously didn't preserve the explicit zeros. 

One might say the correct behavior is now exhibited by scikit-learn (preserving explicit zeros) so this is not really a bug.  However, I am not sure whether the line in the DictVectorizer code referenced above was supposed to preserve explicit zeros or if it was taking advantage of the fancy indexing bug to maintain sparsity. 

In any case, it might at least be worth documenting this change in behavior explicitly so that folks like myself who were relying on this serendipitous behavior know that it would impact downstream code.
user: desilinguist
issue_14713: state: open
title: Simplify spectral clustering solver logic
body: We need to simplify the logic for selecting a solver in ``spectral_clustering``.
See discussion here:
https://github.com/scikit-learn/scikit-learn/issues/10715#issuecomment-369236982
https://github.com/scikit-learn/scikit-learn/pull/14647#issuecomment-521294241
https://github.com/scikit-learn/scikit-learn/pull/10720#issuecomment-518878136
user: amueller
issue_14712: state: open
title: More generic estimator checks
body: #### Description

While trying to roll my own [estimator](https://github.com/raamana/kernelmethods/blob/master/kernelmethods/algorithms.py#L156) as part of [kernelmethods](github.com/raamana), I ran into a failed estimator check [`check_non_transformer_estimators_n_iter`](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/estimator_checks.py#L2283).

This function is not supposed to run the check (as my estimator is derived from SVR) according to code below:

```python
    # These models are dependent on external solvers like
    # libsvm and accessing the iter parameter is non-trivial.
    not_run_check_n_iter = ['Ridge', 'SVR', 'NuSVR', 'NuSVC',
                            'RidgeClassifier', 'SVC', 'RandomizedLasso',
                            'LogisticRegressionCV', 'LinearSVC',
                            'LogisticRegression']

    # Tested in test_transformer_n_iter
    not_run_check_n_iter += CROSS_DECOMPOSITION
    if name in not_run_check_n_iter:
        return
```

However, given my estimator name (`OptimalKernelSVR`) is not actually in the list, it goes ahead and runs the check which is irrelevant to my estimator. To prevent this, I suggest making these checks less specific to sklearn's native estimators and more amenable to external libraries. Some possible solutions are:

1. change ` if name in not_run_check_n_iter:` to

```python
for no_test_name in not_run_check_n_iter:
    if name.lower() in no_test_name.lower():
        return 
```

2. Allow passing some flags to `check_estimator` skip a specific test in [here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/estimator_checks.py#L321)

#### Steps/Code to Reproduce

1. Clone and install [kernelmethods](github.com/raamana/kernelmethods) in dev mode

2. Run this code
```python
from kernelmethods.algorithms import OptimalKernelSVR
from sklearn.utils.estimator_checks import check_estimator
OKSVR = OptimalKernelSVR(k_bucket='light')
check_estimator(OKSVR)
```

#### Versions
```
System:
    python: 3.7.2 (default, Dec 29 2018, 00:00:04)  [Clang 4.0.1 (tags/RELEASE_401/final)]
executable: /Users/Reddy/anaconda3/envs/py36/bin/python
   machine: Darwin-18.6.0-x86_64-i386-64bit
BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: /Users/Reddy/anaconda3/envs/py36/lib
cblas_libs: mkl_rt, pthread
Python deps:
       pip: 19.1.1
setuptools: 41.0.1
   sklearn: 0.21.2
     numpy: 1.15.4
     scipy: 1.1.0
    Cython: None
    pandas: 0.24.2
```
user: raamana
issue_14711: state: open
title: `sklearn/tree/*` refactoring
body: This is the summary of what we have already discussed about the `sklearn/tree/splitter.*` cleaup.

The code uses separate classes for the dense and sparse data, and that mainly is to handle the `pre_sort=True` parameter, which used by the `GradientBoosting*` (not the new `HistGradientBoosting*`).

According to a quick benchmark done by @glemaitre , the `pre_sort=True` parameter gives a 2x speedup, which is insignificant to the speedup provided by the `HistGradientBoosting*`.

There were also other refactoring we could do to speedup the splitter, which we realized while reviewing the `HistGradientBoost*` code. The one I remember is that we could sort the data in the quick sort manner and pass the start and end indices, instead of passing the mask array to each splitter.

also ping @NicolasHug , @ogrisel 
user: adrinjalali
labels: Refactor: Code refactor
issue_14698: state: open
title: [MRG] Adds Correlation Based Feature Selection
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Resolves https://github.com/scikit-learn/scikit-learn/issues/13405


#### What does this implement/fix? Explain your changes.
1. Adds section to user guide.
2. Updates permutation feature importance example with this correlation feature selection.

#### Any other comments?
This should be useful when placed a `Pipeline` in the final estimator when stacking: https://github.com/scikit-learn/scikit-learn/pull/11047

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->
user: thomasjpfan
labels: Needs Decision: Requires decision
issue_14692: state: open
title: should utils.resample accept sample_weight?
body: Working on implementing `sample_weight` for HGBT, I noticed we subsample, but `utils.resample` doesn't accept `sample_weight`.

I think whenever `sample_weight` is provided by the user and we do some sort of sampling, it should be taken into account. WDYT?

ping @ogrisel, @NicolasHug  
user: adrinjalali
labels: Needs Decision: Requires decision
issue_14683: state: open
title: [WIP] Added 2 common tests for sparse matrix densification.
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes  #14483

#### What does this implement/fix? Explain your changes.

I've added 2 common tests that provide estimators with a nominally large sparse matrix and check the memory utilization. Both tests use different approaches and I'm hoping that the reviewers can help me decide which way to go. The current code is WIP and my goal is to get feedback on the approaches. After the reviewers help me decide which way to go, I'll clean up the code more.

The first check is **check_estimator_sparse_data_memory_growth**. As the name indicates, this check aims to assert on the growth in memory requirements for an estimator. It uses the Python builtin profiler hook and checks for peak memory consumption within a context. It then uses the estimator's `fit` and `predict` methods for a baseline input and for 2 inputs that are larger in either the number of rows or number of features. It asserts that the memory requirement does not increase linearly.

The second check is **check_estimator_sparse_data_process_maxrss**. This check uses the `resource` module to check for the max RSS of the process. It does not deal with growth, but just absolute value of the max RSS. I don't know of a way of resetting this max-RSS value for a process; I'm not sure it's possible. Because of this, this check suffers from a few issues: (1) a previous test that uses a lot of memory can raise the max-rss of the process enough that this test completely misses all densifications, (2) If one estimator increases the max-rss sufficiently and causes a failure, it can mask the failures of succeeding checks for estimators.

In these 2 ways, I wanted to not only show 2 techniques for measuring memory consumption, but also 2 ways of asserting on memory (growth vs. absolute).

I also wanted to mention some additional aspects of the 2 approaches:

**check_estimator_sparse_data_memory_growth**

* Adding a profiling hook causes the code to slow down during the context of the memory profiler.
* To do cross-platform memory consumption measurement, I've added a third-party dependency (`psutil`). Maybe if the reviewers think that that's a problem, I can look into removing it.
* To check whether the memory growth is sub-linear, I probably need to measure at more than 2 points per axis. That would probably cause this check to take a while. I'm not sure what kind of runtime is acceptable.

**check_estimator_sparse_data_process_maxrss**

* I'm not sure how one would select an absolute threshold for memory. Probably better to go with the growth idea from the check above.
* As I mentioned, using max-rss seems like a flawed idea because test isolation is an issue.

##### Results

There are a fair number of failures using the memory growth idea. If the idea receives positive feedback, I'd be happy to look into it. A lot of these failures are genuine and would probably be fun to fix 😄 
Also note that it takes 85.19 seconds to run the growth tests and that 34 out of 162 tests fail.

On the other hand, the RSS test is a lot faster at 24.64 seconds. It also has only 2 failures, but that doesn't actually mean anything because it's based on an arbitrary threshold that I've added to it.

Note that the regular sparse data check (`check_estimator_sparse_data`) takes only 11.04 seconds to run on my machine. So the memory tests are going to add a fair amount of time to the CI pipeline.

#### Any other comments?

There are more approaches to solving this issue:

1. I could just create a very very large sparse matrix and see if it blows up the process. As was mentioned by @rth, this would be a very annoying dev-experience.
2. I could try to mock methods in sparse matrices that densify it. This approach is going to get very complex very fast as we try to keep track of the internal methods of sparse matrices that could detect this. Also something as simple as cloning the matrix could thwart this and to avoid that, we'll have to add further complexity. I'm not a fan of this approach.

Obviously the `LeakyEstimator` exists only for development purposes in case any reviewers want to try running the tests themselves. I'll delete it if and when we're ready to merge. 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: chhabrakadabra
issue_14654: state: open
title: [MRG] ENH add support for multiclass-multioutput to ClassifierChain
body: #### Reference Issues/PRs
Fixes #9245
Fixes #13338
Fixes #13339

#### What does this implement/fix? Explain your changes.
`predict_proba` and `decision_function` of `ClassifierChain` retuns now a list of `n_outputs` arrays, each of  shape `(n_samples, n_classes)`. This format is consistent with that of `MultiOutputClassifier`. 

#### Any other comments?
`multiouput_only` tag has now been removed. I wonder whether we should keep the `_skip_test` tag. Tests pass even when this tag is removed. 

This change will break backward compatibility, since the functions previously returned arrays rather than lists. Any ideas on how to handle this?

user: agamemnonc
labels: Needs Decision: Requires decision
Breaking Change: Issue resolution would not be easily handled by the usual deprecation cycle.
issue_14648: state: open
title: adding multilabel capability to voting classifier and bagging classifier
body: #### Description
There is so many classifiers that  can not use multilabel, but why voting classifier and bagging classifier can not use multilabel? the base classifier in the voting and bagging classifer already support multilabel.
user: cgcodeboy
issue_14636: state: open
title: ENH: Add a variable "copy" to perform in-place laplacian calculation in spectral clustering
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

superseded  #8842
closes  #8842
closes #8834


#### What does this implement/fix? Explain your changes.

Add a variable named "copy" to perform in-place laplacian calculation in spectral clustering

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: mehrnazmo
issue_14612: state: open
title: include correlation matrix function at covariance estimators
body: hi folks, i'm using Ledoit Wolf algorithm, and i need correlation matrix instead of covariance

could be possible include this function at sklearn covariance algorithms?

```

def get_corr_from_cov(covmat):
    d = np.diag(np.sqrt(np.diag(covmat)))
    invd = np.linalg.inv(d)
    return invd.dot(covmat).dot(invd)

```

thanks!
user: rspadim
issue_14607: state: open
title: Add target for plot_roc_curve
body: Similar to how in plot partial dependence we have:

```py
    target : int, optional (default=None)
        - In a multiclass setting, specifies the class for which the PDPs
          should be computed. Note that for binary classification, the
          positive class (index 1) is always used.
        - In a multioutput setting, specifies the task for which the PDPs
          should be computed
        Ignored in binary classification or classical regression settings.
```
user: thomasjpfan
labels: Hard: Hard level of difficulty
issue_14569: state: open
title: Adding L1 Isotonic Regression, want to contribute
body: Hi,
Currently sklearn.isotonic.IsotonicRegression only supports L2 loss.
L1 support is missing.
I have implemented and tested algorithm from this paper: http://drops.dagstuhl.de/opus/volltexte/2018/10027/pdf/OASIcs-SOSA-2019-1.pdf
Question:
Would the community be interested in the adding this code to sklearn's codebase?
if what is the best way to proceed.
* I am a full time Google employee and contribution depends on approval.

Liran Katzir.

user: lirankatzir
labels: Needs Decision: Requires decision
issue_14560: state: open
title: [MRG] Binned regression cv
body: Follow up on #4757 with a much simplified implementation.
Fixes #4757.

If anyone has an idea for a better/shorter name I'm all ears!
user: amueller
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_14559: state: open
title: CountVectorizer sets self.vocabulary_ in transform
body: Right now CountVectorizer sometimes sets ``self.vocabulary_`` outside of ``fit``. We usually prohibit this, but the common tests haven't reached the vectorizers yet.
user: amueller
issue_14552: state: open
title: Incorrect density values while implementing KDE on point data(X,Y) (scikit-learn)
body: I am estimating point density, where I am having point coordinates in degrees, I need a raster(over the extent of these points) which provides a density estimation on each cell.

I have tried on R but I have not got any success as the extent of my point data is very large. I am trying scikit-learn package in python, but stucked with the implementation.

    import numpy as np
    import matplotlib.pyplot as plt
    from sklearn.neighbors import KernelDensity
    import matplotlib.pyplot as plt
    import gdal
    import os
    import ogr
    import osr
    from sklearn.cluster import DBSCAN

    def density_Viz(str_InputSHP):
        driver = ogr.GetDriverByName('ESRI Shapefile')
        ds = driver.Open(str_InputSHP)
        layer = ds.GetLayer(0)
        lst_train = []
        lst_coord = []
        for feat in layer:
            geom = feat.GetGeometryRef()
            X = geom.GetX()
            Y = geom.GetY()
            xy = (X, Y)
            lst_coord.append(xy)
            lst_train.append([X, Y])

        # val=getKernelDensityEstimation(lst_coord)
        lst_train = np.array(lst_train)
        lat_lon_val = lst_train.copy()
        lst_train *= np.pi / 180.
        # create grid
        (dblXmin, dblXmax, dblYmin, dblYmax) = layer.GetExtent()
        pixelWidth = 0.00082
        Nx = int((dblXmax - dblXmin)/pixelWidth)
        Ny = int((dblYmax - dblYmin)/pixelWidth)
        xmin = dblXmin + 0.0008
        xmax = xmin + (Nx * pixelWidth)
        ymin = dblYmin + pixelWidth
        ymax = ymin + (Ny * pixelWidth)
        # x coordinates of the grid cells
        xgrid = np.arange(xmin, xmax, pixelWidth)
        # y coordinates of the grid cells
        ygrid = np.arange(ymin, ymax, pixelWidth)

        X, Y = np.meshgrid(xgrid, ygrid)
        xy = np.vstack([X.ravel(), Y.ravel()]).T
        xy *= np.pi / 180.

        kde = KernelDensity(bandwidth=(0.0032*np.pi/180), metric='haversine',
                            kernel='epanechnikov', algorithm='ball_tree', breadth_first=False).fit(lst_train)

        val = kde.score_samples(lst_train)
        val1 = kde.score_samples(xy)

        print(np.amax(val))  # getting 16, expected value is nearly 70
        print(np.amin(val))  # getting 13
        print(np.amax(val1))  # getting 16
        print(np.amin(val1))  # getting -Inf
        print("Done")
user: GISConsortium
issue_14524: state: open
title: [Feature] Adding a Group And Label Kfold Split Method to Model Selection
body: [Feature] This commit implements an additional model selection split strategy names LabelAndGroupKFold which treats the label and group combination as something that can not be split across multiple folds, while at the same time attempting to evenly distribute labels across folds so that each fold has a similar amount of each label type. This is a extension of the GroupKfold, which does not attempt to evenly distribute the classes, only the groups.

user: cdknorow
issue_14483: state: open
title: Add common test for huge sparse matrix
body: I think we should add a common test that checks that results with a giant sparse matrix are reasonable, to ensure we don't make them dense anywhere.
It should be pretty easy to make sure the nominal shape of the sparse matrix in the sparse matrix test is really big, so that any densifying operation would make the tests time out.
user: amueller
labels: module:test-suite: everything related to our tests
issue_14481: state: open
title: Read-only X + copy=False Fails for most estimators 
body: calling fit or transform fails for most estimators having a copy or copy_x or copy_X parameter (see #14480).

We could
- raise an error with a more informative message.
- emit a warning and copy the data
- copy data silently and say in the docstring that even if copy=False a copy will still be made if X is read-only (we already do that for other reasons)

WDYT ?
user: jeremiedbb
issue_14473: state: open
title: Inconsistent intercept_ shape between regressors
body: Sometimes we return float, sometimes we return numpy array. Seems unfriendly, since numpy array is different from float in many ways.
```py
from sklearn.datasets import load_boston
from sklearn.linear_model import LinearRegression, Ridge, SGDRegressor
from sklearn.svm import LinearSVR
X, y = load_boston(return_X_y=True)
regs = [LinearRegression(), Ridge(), SGDRegressor(), LinearSVR()]
for reg in regs:
    reg.fit(X, y)
    if isinstance(reg.intercept_, np.float):
        print(reg.__class__.__name__, "intercept shape: float")
    else:
        print(reg.__class__.__name__, "intercept shape:", np.shape(reg.intercept_))
```
```
LinearRegression intercept shape: float
Ridge intercept shape: float
SGDRegressor intercept shape: (1,)
LinearSVR intercept shape: (1,)
```
or maybe it's trivial and we can ignore it?
user: qinhanmin2014
issue_14468: state: open
title: Support Mini-Batch gradient descent
body: Currently, we support batch gradient descent (e.g., LogisticRegression), and stochastic gradient descent (e.g., ``SGDClassifier/SGDRegressor``), but we do not support Mini-Batch gradient descent (``SGDClassifier/SGDRegressor.partial_fit`` is not Mini-Batch gradient descent).
Mini-Batch gradient descent can support partial_fit (compared with batch gradient descent) and can sometime converge to better results (compared with stochastic gradient descent).
user: qinhanmin2014
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_14444: state: open
title: Provide an option to consider class prior in ComplementNB
body: Currently, in ComplementNB, we estimate class prior but do not use it. I think we can provide an option to consider class prior in ComplementNB (like other naive bayes algorithms in scikit-learn). Reasons:
(1) In the original paper, when proposing ComplementNB, the authors actually take class prior into consideration (see Section 3.1). When proposing the detailed implementation, the authors "use a uniform prior estimate for simplicity", because they think that the class probabilities tend to be overpowered by class prior.
(2) This will make ComplementNB consistent with other naive bayes algorithms in scikit-learn, which is beneficial if we want to implement GeneralNB in the future.
(3) Current API design of ComplementNB seems awkward, i.e., we expose unused parameters (class_prior) to users.

Some simple benchmarks on 20 newsgroups (fetch_20newsgroups_vectorized):
MultinomialNB  training set acc: 0.8533 testing set acc: 0.7159
MultinomialNB + class prior  training set acc: 0.8439 testing set acc: 0.7053
ComplementNB  training set acc: 0.9498 testing set acc: 0.8318
ComplementNB + class prior  training set acc: 0.9156 testing set acc: 0.8043
user: qinhanmin2014
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_14440: state: open
title: Add Supervised Latent Dirichlet Allocation (sLDA) to the API
body: Hello,

This is more a feature request than an issue/ticket, but I believe it would be great if scikit learn implemented supervised Latent Dirichlet Allocation (sLDA) and/or other topic modelling approaches. I saw some of the classes in this [repo](https://github.com/Savvysherpa/slda), and I think the community would benefit from having something like those classes in skleaen.

Thanks,
Juan
user: juanluislm
issue_14437: state: open
title: Faster GPs using HODLR solver
body: The scikit-learn implementation of Gaussian Processes (GPs) scales as *O(n^3)* where *n* is the number of data points. Other packages such as [celerite](https://github.com/dfm/celerite) or [george](https://github.com/dfm/george) offer the [HODLR solver](https://arxiv.org/abs/1403.6015) which scales as *O(n log^2 n)*. This allows to use GPs for datasets with e.g. 10k points in seconds, instead of minutes. See the speed comparison figure at the bottom of [this page](http://dfm.io/george/dev/tutorials/scaling/).

I would love to have that speed-up in scikit-learn as well. Both of the packages mentioned are under the MIT license. Is it possible to implant this functionality into scikit-learn? [This implementation is in C++](https://github.com/dfm/george/blob/master/george/solvers/_hodlr.cpp)  (174 SLOC) with [Python bindings](https://github.com/dfm/george/blob/master/george/solvers/hodlr.py) (34 SLOC).
user: hippke
labels: Hard: Hard level of difficulty
issue_14419: state: open
title: Formulate for _compute_lower_bound in gaussian_mixture
body: Hello,

I am having trouble following the computation for [**_compute_lower_bound**](https://github.com/scikit-learn/scikit-learn/blob/7813f7efb5b2012412888b69e73d76f2df2b50b6/sklearn/mixture/gaussian_mixture.py#L686) in gaussian_mixture.py. My understanding is that this should be the expected likelihood of the current estimated parameters. I'm using the Wikipedia article on [Expectation Maximization](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm#E_step) as a reference, specifically the second to last line in this [equation](https://wikimedia.org/api/rest_v1/media/math/render/svg/03b12df3e8225f53ef07fd5cf6b255f76040c652). My interpretation of that expression is that the expected log likelihood should be the sum over all classes and all data points of P(Zi=j|Xi=xi;theta)*log(P(xi,zi|theta)). Mapping these terms to existing variables:

P(Zi=j|Xi=xi;theta) = exp(**log_resp**)
log(P(xi,zi|theta)) = **weighted_log_prob**

So that the lower-bound (expected log likelihood) should be
`np.mean(np.sum(np.exp(log_resp) * weighted_log_prob, 1))`

where the mean normalizes by number of data points. But, when I follow the propagation of variables for how it is currently computed:

[base.py L503](https://github.com/scikit-learn/scikit-learn/blob/7813f7efb5b2012412888b69e73d76f2df2b50b6/sklearn/mixture/base.py) : In `_estimate_log_prob_resp`, it calls `weighted_log_prob = self._estimate_weighted_log_prob(X)`, calculates `log_prob_norm = logsumexp(weighted_log_prob, axis=1)` and return `log_prob_norm` as first return element.

[base.py L295](https://github.com/scikit-learn/scikit-learn/blob/7813f7efb5b2012412888b69e73d76f2df2b50b6/sklearn/mixture/base.py#L295) : `_e_step` receives `log_prob_norm, log_resp = self._estimate_log_prob_resp(X)` and returns `np.mean(log_prob_norm)` as first element of return.

[base.py L243](https://github.com/scikit-learn/scikit-learn/blob/7813f7efb5b2012412888b69e73d76f2df2b50b6/sklearn/mixture/base.py#L243) : `lower_bound = self._compute_lower_bound(log_resp, log_prob_norm)` with `log_prob_norm, log_resp = self._e_step(X)` obtained a few lines above.
[gaussian_mixture.py L686](https://github.com/scikit-learn/scikit-learn/blob/7813f7efb5b2012412888b69e73d76f2df2b50b6/sklearn/mixture/gaussian_mixture.py#L686) : `_compute_lower_bound` returns `log_prob_norm` unchanged

Putting all that together: `_compute_lower_bound` returns `np.mean(logsumexp(weighted_log_prob, axis=1))`

which differs from mapped from the Wiki article above:
`np.mean(np.sum(np.exp(log_resp) * weighted_log_prob, 1))`

Both expressions yield similar values in my example cases, but the later expression is a little lower.
user: mattbrown11
issue_14388: state: open
title: MNT Refactor the three base scorers into _BaseScorer
body: This PR refactors the three base scorers (_PredictScorer, _ProbaScorer, _ThresholdScorer) into _BaseScorer. These are all private classes so we can remove them directly.
Honestly I'm not sure whether we want this (vote +0 for myself :))

- Advantages:
  * avoid duplicate code, easier to maintain
  * avoid inconsistency between different base scorers (e.g.,  previously, for binary y_true, _ProbaScorer requires score function to accept 2d y_pred, but _ThresholdScorer requires score function to accept 1d y_pred, see #14318)

- Disadvantages:
  * Maybe more difficult if we want to add more base scorers?
user: qinhanmin2014
labels: Needs Decision: Requires decision
issue_14384: state: open
title: Add feature distributed data loading for load_svmlight_file()
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
This PR allows for sampling a subset of features [start_feature, end_feature) from a libsvm file without loading all the features.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: anxuthu
issue_14345: state: open
title: Add support for pLSA
body: Currently scikit-learn has implementations of NMF and LatentDirichletAllocation. In practice pLSA sits between the two, with a simple and easy to explain computational approach that can run very quickly (faster than NMF with KL loss, slightly slower than NMF with Frobenius loss), but the interpretable probabilistic model similar to LDA. The pLSA paper is from 2001, and has over 2500 citations, so it certainly seems well established.

I have implemented pLSA using numba (https://github.com/lmcinnes/enstop/blob/master/enstop/plsa.py), and the implementation is very straightforward. I believe it would be easy to convert this to Cython for potential inclusion in scikit-learn (ultimately only the e-step, m-step and inner loop need to be cythonized for performance). I would be happy to work on an implementation if there was interest in seeing pLSA included.
user: lmcinnes
issue_14338: state: open
title: IterativeImputer not converging (at all)
body: Observed in #14330: the IterativeImputer doesn't converge, as a matter of fact, the convergence criterion doesn't seem to go down with iterations.
To me that indicates that either there's an issue with the implementation, an issue with the metric, or that our stopping criterion is not suitable.

Were there examples on which it converged? It would be nice to plot the convergence but we don't allow callbacks like that. But just printing it on https://github.com/scikit-learn/scikit-learn/blob/master/examples/impute/plot_iterative_imputer_variants_comparison.py shows that it doesn't go down.
user: amueller
labels: Moderate: Anything that requires some knowledge of conventions and best practices
High Priority: High priority issues and pull requests
issue_14328: state: open
title: Stale scikit-learn datasets can throw pickling errors
body: #### Description

Loading a dataset that has been downloaded to ~/scikit_learn_data some time in the past can throw unpickling errors after updating scikit-learn. Possible solutions might be: 

- clearing the dataset cache after updating 
- including the scikit-learn version used to serialize the dataset objects 
- serializing with something besides `pickle`

Deleting the data folder resolves this issue.

#### Steps/Code to Reproduce

```
python examples/impute/plot_iterative_imputer_variants_comparison.py
```

#### Expected Results

No error is thrown.

#### Actual Results

An unpickling error is thrown when trying to find the `_joblib` module in `sklearn.externals`.

```
Traceback (most recent call last):
  File "examples/impute/plot_iterative_imputer_variants_comparison.py", line 61, in <module>
    X_full, y_full = fetch_california_housing(return_X_y=True)
  File "/Users/dillon/githubpackages/scikit-learn/sklearn/datasets/california_housing.py", line 133, in fetch_california_housing
    cal_housing = _refresh_cache([filepath], 6)
  File "/Users/dillon/githubpackages/scikit-learn/sklearn/datasets/base.py", line 930, in _refresh_cache
    data = tuple([joblib.load(f) for f in files])
  File "/Users/dillon/githubpackages/scikit-learn/sklearn/datasets/base.py", line 930, in <listcomp>
    data = tuple([joblib.load(f) for f in files])
  File "/Users/dillon/anaconda/envs/sklearn-dev/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 598, in load
    obj = _unpickle(fobj, filename, mmap_mode)
  File "/Users/dillon/anaconda/envs/sklearn-dev/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 526, in _unpickle
    obj = unpickler.load()
  File "/Users/dillon/anaconda/envs/sklearn-dev/lib/python3.7/pickle.py", line 1085, in load
    dispatch[key[0]](self)
  File "/Users/dillon/anaconda/envs/sklearn-dev/lib/python3.7/pickle.py", line 1373, in load_global
    klass = self.find_class(module, name)
  File "/Users/dillon/anaconda/envs/sklearn-dev/lib/python3.7/pickle.py", line 1423, in find_class
    __import__(module, level=0)
ModuleNotFoundError: No module named 'sklearn.externals._joblib'
```

#### Versions

```
System:
    python: 3.7.3 (default, Mar 27 2019, 16:54:48)  [Clang 4.0.1 (tags/RELEASE_401/final)]
executable: /Users/dillon/anaconda/envs/sklearn-dev/bin/python
   machine: Darwin-17.7.0-x86_64-i386-64bit

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: /Users/dillon/anaconda/envs/sklearn-dev/lib
cblas_libs: mkl_rt, pthread

Python deps:
       pip: 19.1.1
setuptools: 41.0.1
   sklearn: 0.22.dev0
     numpy: 1.16.4
     scipy: 1.2.1
    Cython: 0.29.11
    pandas: 0.24.2
matplotlib: 3.1.0
    joblib: 0.13.2
```


user: deniederhut
issue_14314: state: open
title: Use of X.dtype when it is not float
body: A fairly common pattern in the scikit-learn code is to create intermediary arrays with `dtype=X.dtype`. That works well, as long as `X = check_arrays(X)` was run with a float dtype only, or in other words when `X.dtype` is not int.

When `X.dtype` is int, `check_array(X)` with default parameters will pass it though, and then all intermediary objects will be of dtype int. 

For instance,
```py
import numpy as np
from sklearn.cluster import MiniBatchKMeans

X = np.array([[1, 2], [1, 4], [1, 0],
              [4, 2], [4, 0], [4, 4],
              [4, 5], [0, 1], [2, 2],
              [3, 2], [5, 5], [1, -1]])
# manually fit on batches
kmeans = MiniBatchKMeans(n_clusters=2,
                         random_state=0,
                         batch_size=6)
kmeans.partial_fit(X[0:6,:])
```
(taken from the [MiniBatchKMeans docstring](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.MiniBatchKMeans.html)) 
will happily run in the integer space where both `sample_weight` and `cluster_centroid_` will be `int`. Discovered as part of https://github.com/scikit-learn/scikit-learn/pull/14307

Another point, that e.g. in linear models, `check_array(..., dtype=[np.float64, np.float32])` will only be run if `check_input=True` (default). Meaning that when `check_input=False` it might try to create a liner model where coefficients are an array of integers, and unless something fails due to a dtype mismatch the user will never know. I think we should always check that `X.dtype` is float, even when `check_input=False`. I believe the point of that flag was to avoid expensive checks and this doesn't cost anything.

In general any code that uses `X.dtype` to create intermediary arrays should be evaluated as to whether we are sure the dtype is float (or that ints would be acceptable).

Might be a good sprint issue, not sure.
user: rth
issue_14311: state: open
title: Add common test for constant features
body: There should be a common test that estimators work with constant features or raise a reasonable error message. Also see #13609.
user: amueller
issue_14306: state: open
title: Multicore scalability of the Histogram-based GBDT
body: I observed that our OpenMP based GBDT does not scale well beyond 4 or 8 threads.

@SmirnovEgorRu who already contributed scalability improvements to xgboost sent me the following analysis by email, reproduced here with his permission:

I had initial looking at histogram building functions and observed a few things:

Scikit-learn’s code spends too much time in np.zeroes() functions. It is used for creation and filling histograms by zeroes. Theoretically – it is a cheap operation, but this is sequential and uses not native code. I replaced usage of np.zeroes() by np.empty() and scaling was improved, for max number of threads performance improvement – around 30%.

For low decision-tree nodes - # of rows can be very small < 1000. In this case creation of parallel loop brings more overhead than value. Similar issue was covered here - https://github.com/hcho3/xgboost-fast-hist-perf-lab/issues/15, to understand background of the issue – you can read readme in the repository. I hope, you will find smth interesting for you there.

Link above also says about impact of conversion float <-> double into performance. I have looked at Scikit-learn’s assembler of histogram building and observed conversions too. I haven’t tested, but removing of this conversion can help.

If exclude np.zeroes() – building of histograms scales okay if we use only one processor. If we say about multi-socket systems (NUMA) – it scales very poor.

## Here I can recommend you:

- Split bin-matrix to <# of numa –nodes> parts by rows and allocate memory on each numa-node separately.

- Pin threads to numa-nodes. Each thread should use bin-matrix only from pinned numa-node.


/cc @NicolasHug @jeremiedbb @adrinjalali @glemaitre @pierreglaser 
user: ogrisel
issue_14291: state: open
title: Use SVD of covariance matrix instead of X if #features < #data
body: Correct me, if I am wrong, but the idea of a PCA is to decorrelate a given data sample X, right? Typically, this is achieved by diagonalizing the covariance matrix C of X, i.e. finding an orthonormal matrix R, such that (R^T C R) becomes diagonal. **If** X is centralized, this is equivalent with the SVD of X though, since C = (X^T X) / (n-1) in this case. Given X = (U S V^T) is the result of the SVD, then V is equivalent with R (hence X -> XR = XV is decorrelated). 

My point here is, that if X is a (m x n) matrix with (m > n, i.e. more data (rows) than features), the former approach is much cheaper than the latter, since finding the covariance matrix roughly scales like O(mn^2) and the SVD (aka diagonalization) of the square (n x n) covariance matrix is basically for free for small n. This makes it way more efficient than the (bi)diagonalization via SVD of a full-blown (m x n) matrix (i.e. X), which is O(mn^2 + nm^2). (Note the m^2!)

My proposal: Replace the SVD of the (m x n) matrix X with a SVD of the (n x n) covariance matrix of X. You can then even relax the centralization requirement of X since the proposed solution works always. (N.B: The ordering of the PCA components is preserved if you use the SVD of the covariance matrix.)
user: avitase
issue_14269: state: open
title: Better selection of the default solver for Ridge based on the shape of data
body: As reported in the discussion of #13923, `Ridge(solver="cholesky")`  can be significantly faster than the default setting for tall dense X.

We should explore the opportunity to use a better heuristic to select the default solver based on the shape of the input data.

One would need to run some benchmarks to design a better heuristic and also evaluate the (lack of) impact of `alpha` for the choice of the default solver.

Related issue for Ordinary Least Squares (no penalty): #14268
user: ogrisel
issue_14268: state: open
title: Add option to use different solver to LinearRegression
body: As reported in #13923 the currently used `scipy.linalg.lstsq` can be significantly slower than `Ridge(solver="cholesky", alpha=0)` for tall dense X.

We should also call `scipy.linalg.lstsq` with `check_finite=False` as we already do input validation in `fit`.

Also the `scipy.linalg.lstsq` function has an optional `lapack_driver` that accepts the following options: ``'gelsd'``, ``'gelsy'``, ``'gelss'``.  The default is``'gelsd'``. Maybe we should expose the others in our API and benchmark them to see if it would make sense to change the default.

Related issue for Ridge: #14269
user: ogrisel
issue_14266: state: open
title: [WIP] FIX make sure sample_weight has an effect on predictions
body: Alternative/complement to #14246 

Check that `sample_weight` change the prediction of the estimator. In case this is not true, it could be because of a bug.
user: glemaitre
labels: Needs Decision: Requires decision
issue_14257: state: open
title: Feature request: Group aware Time-based cross validation
body: Basically combining `TimeSeriesSplit` with the `Group` awareness of other CV strategies such as `GroupKFold`.

I think it's a good first issue for first time contributors that are already familiar with the existing cross validation tools in scikit-learn:

https://scikit-learn.org/stable/modules/cross_validation.html

Source code is here:

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/model_selection/_split.py
user: ogrisel
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_14256: state: open
title: Kappa score cannot return 1
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->

Cohen kappa score cannot return 1 in fully matched.

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

```python:
from sklearn.metrics import cohen_kappa_score
a = [1, 1, 1, 1, 1]
b = [1, 1, 1, 1, 1]

print(cohen_kappa_score(a, b))
# return nan
print(cohen_kappa_score(a, b, labels=[1, 2, 3]))
# return nan
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->

Theoretically, kappa must return 1.0 in fully matched.

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->

nan

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->

```
System:
    python: 3.7.3 (default, Mar 27 2019, 09:23:39)  [Clang 10.0.0 (clang-1000.11.45.5)]
   machine: Darwin-17.7.0-x86_64-i386-64bit

BLAS:
    macros: NO_ATLAS_INFO=3, HAVE_CBLAS=None
  lib_dirs:
cblas_libs: cblas

Python deps:
       pip: 18.1
setuptools: 40.6.2
   sklearn: 0.20.2
     numpy: 1.16.1
     scipy: 1.2.1
    Cython: None
    pandas: 0.24.2
```

<!-- Thanks for contributing! -->

user: hayata-yamamoto
issue_14253: state: open
title: Use different `estimator` for different features in IterativeImputer
body: An idealised IterativeImputer would allow some features to be predicted as categorical variables (with a classifier) and others to be predicted as continuous variables (with a regressor) or even other specific distributions.

We could consider an interface to IterativeImputer that would allow users to specify the estimator used for a particular column selector, much like ColumnTransformer does with transformers.

On the other hand we may decide that this adds rarely needed power with excessive complexity.
user: jnothman
labels: Hard: Hard level of difficulty
issue_14248: state: open
title: RFC Common internal optimizers API
body: Linear models and neural nets use a number of optimizer functions/solvers e.g. (`fmin_l_bfgs_b` (scipy), `newton_cg` etc). For some solvers (e.g. `lbfgs`) the individual estimators then needs to check that the convergence happened, and print the appropriate warning. This is done more or less well and sometimes differently depending on the estimator (e.g. https://github.com/scikit-learn/scikit-learn/pull/9274#discussion_r298675005 recently). Other solvers check the convergence internally.

scipy has done some work to unify optimizers with the [scipy.optimize.minimize](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize) interface. I think it might make sense to use that API when possible (I haven't done a detailed analysis yet to detect possible issues), and check the resulting [`OptimizeResult`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html#scipy.optimize.OptimizeResult) object. 

This should allow to,
 - remove some redundant code in estimators by reusing the same helper functions.
 - increase confidence that we are properly checking for the successful convergence
 - although it doesn't directly address it, having a common solvers API might help if we want to tackle the issue of determining equivalent stopping criterion (`tol`) between solvers. 
 - would help to support third party solvers in the future.

TBH, I'm not sure how much of this can really be generalized outside of lbfgs, that can use a few helper functions to check convergence.  

Also related to https://github.com/scikit-learn/scikit-learn/pull/9405 that includes a Iteratively Reweighted Least Squares (IRLS) solver.

Comments are welcome.
user: rth
issue_14236: state: open
title: `test_gradient_and_hessian_sanity` fails in 32bit
body: [Link to test failure](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=4574). `test_gradient_and_hessian_sanity` fails in 32bit:

```py
        for hists, indices in (
                (hists_parent, sample_indices),
                (hists_left, sample_indices_left),
                (hists_right, sample_indices_right)):
			...
            gradients = hists['sum_gradients'].sum(axis=1)  # shape = (n_features,)
            expected_gradient = all_gradients[indices].sum()  # scalar
			...
>           assert np.allclose(gradients, expected_gradient)
E           assert False
E            +  where False = <function allclose at 0xf7209adc>(array([  2.68156159e+154,   8.09900937e+000,  -5.20938628e+238,\n         2.01738272e+118,  -2.20855883e+071,   4.74963...287e+000,\n         9.24455781e+000,   6.38250258e-001,  -3.41402339e+243,\n        -9.90611303e-001,   6.27710174e+057]), 26.061348)
E            +    where <function allclose at 0xf7209adc> = np.allclose

```

Discovered in https://github.com/scikit-learn/scikit-learn/pull/14225
user: thomasjpfan
issue_14229: state: open
title: Make valid pairwise distance metrics (i.e. _VALID_METRICS variable) public
body: The list of available distance metrics is currently stored in a private list called `_VALID_METRICS` (https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/pairwise.py#L1237). It'd be awesome to make this available as a public variable or function. 

I'm aware of `distance_metrics` function (https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/pairwise.py#L1147), but the list there is not complete as scipy functions are not listed.

Here is a use case how this might be useful: https://github.com/lmcinnes/umap/pull/259
user: gokceneraslan
issue_14228: state: open
title: Better documentation for n_jobs
body: Sort of a follow-up to https://github.com/scikit-learn/scikit-learn/issues/10415.

The great majority of the docstrings for `n_jobs` is something like "number of jobs to run in parallel".

It would be much more helpful for users to understand where and how parallelization is actually applied.

For example for random forest: "Each tree is built in parallel" would be a simple yet significant improvement.
user: NicolasHug
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_14198: state: open
title: Advanced API
body: In a few places we've mentioned certain functions which we would like to have, but they don't necessarily fit the usual `fit`/`transform`/`predict` pattern.

For instance, @jnothman mentioned having an `add` function for the `ColumnTransformer`, which we can also have for the `Pipeline`.

We also talked about a `prune_tree` function in #14038.

One concern seems to be that we would like to keep the API very simple and easy, which I agree with. But it doesn't have to limit us from having more ad-hoc functions, which we could fit in a separate section in the docs and tag them as _advanced_.

I'm not sure how we could handle that with sphinx, but what I'm proposing is to tag those functions as _advanced_, and have sphinx render them in an _"advanced"_ section bellow the other functions. This way they would not interfere with the usual experience of a new user who's reading the docs, and yet it would enable us to introduce some rather useful methods.

I may be missing some historical discussion on this topic though, sorry for that.
user: adrinjalali
labels: Hard: Hard level of difficulty
issue_14191: state: open
title: BaggingClassifier vs LinearSVC training slow
body: 
#### Description

BaggingClassifier with base_estimator=LinearSVC(), n_estimators=10, n_jobs=10, 
max_samples=0.1 takes the same time to train as LinearSVC().

#### Steps/Code to Reproduce

```python
from sklearn.svm import LinearSVC
from sklearn.ensemble import BaggingClassifier

base_estimator = LinearSVC(random_state=42, tol=1e-6)
n_estimators = 10
max_samples = 1.0 / n_estimators
clf = BaggingClassifier(base_estimator, n_estimators=n_estimators,
                        n_jobs=10, max_samples=max_samples)
clf.fit(X_trn, y_trn)
```

#### Expected Results

I expected it to train about 10 times faster

#### Actual Results

Trains for the same amount of time as its base estimator LinearSVC(). 
Produces exactly the same accuracy as LinearSVC(), which is also strange.

When I monkey-patched base estimator's fit() method:

```python

# Pretend that base_estimator.fit() doesn't support "sample_weight"
def fit_no_sample_weight(estimator, X, y):
    return estimator._original_fit(X, y)

base_estimator._original_fit = base_estimator.fit
base_estimator.fit = fit_no_sample_weight.__get__(base_estimator, LinearSVC)
```

it trained as expected: about 10 times faster and accuracy was lower.

#### Versions

System:
    python: 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)  [GCC 7.3.0]
executable: /foo/anaconda3/bin/python
   machine: Linux-3.16.36begun-x86_64-with-centos-7.3.1611-Core

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: /foo/anaconda3/lib
cblas_libs: mkl_rt, pthread

Python deps:
       pip: 19.0.3
setuptools: 40.8.0
   sklearn: 0.21.2
     numpy: 1.16.2
     scipy: 1.2.1
    Cython: 0.29.5
    pandas: 0.24.1
user: vadamoto
issue_14186: state: open
title: Data type of degree parameter for polynomial kernels
body: According to the docs, the `degree` parameter for a `metrics.pairwise.polynomial_kernel` (and, e.g., also for a `decomposition.KernelPCA` or a `svm.SVR`) module should be `int` ([[1]](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.polynomial_kernel.html#sklearn.metrics.pairwise.polynomial_kernel), [[2]](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.KernelPCA.html#sklearn.decomposition.KernelPCA), [[3]](https://scikit-learn.org/dev/modules/generated/sklearn.svm.SVR.html#sklearn.svm.SVR)). 

Mathematically, fractional values for the degree make things a little more complicated; however, the resulting kernels seem to enjoy some [interesting properties](https://link.springer.com/content/pdf/10.1007%2FBFb0026684.pdf), because they effectively map into an infinitely-dimensional space (just like, e.g., RBF kernels). Furthermore, from a technical point of view, the actual [`polynomial_kernel`](https://github.com/scikit-learn/scikit-learn/blob/7813f7efb/sklearn/metrics/pairwise.py#L855) function never really makes use of the assumption that `degree` is an integer anyway (the `**=` operation works for both floats and integers), and in practice, a `float` value for `degree` already seems to work just fine.

I was therefore wondering if there is a particular reason why the documentation calls for an integer for the `degree` parameter, or whether this is unintentional and the docs can / should be updated? :) In the latter case, I'd be happy to try and submit a PR.
user: timothygebhard
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_14181: state: open
title: Make AIF360 default bias checker and mitigator in scikit-learn
body: [AIF360](https://github.com/IBM/AIF360) is a bias detection and mitigation tool. AIF360 has the concept of pre-processing, in-processing and post-processing algorithms. 

**Initial Request:**
Initial request to make AIF360 scikit-learn compliant came in AIF360 from Adrin Jalali @adrinjalali  when he opened an issue to make in-processing compatible with scikit-learn estimators, and scorers should also be similar in both.
https://github.com/IBM/AIF360/issues/58

In his fork, he created a tutorial for building Fair AI Models using AIF360 by workarounds, where he had to break from scikit-learn pipeline paradigms.
https://github.com/adrinjalali/aif360_tutorial/blob/master/building_fair_AI_models.ipynb

**Follow-On:**
IBM Research started a follow-on, and Samuel Hoffman (@hoffmansc) responded by a subsequent course of action which can be followed:

- [ ] Move changes to a single branch in AIF360
- [ ] Start by converting AIF360 Metrics to sklearn scorers using workarounds, if needed
- [ ] Convert as many bias mitigation algorithms to Estimators as possible
- [ ] Push for sklearn PRs mentioned at bottom, and additional necessary functionality
- [ ] Swap out code in large chunks (e.g. all metrics at once) in the master branch to ensure a useable state at each release


The SLEPs (scikit-learn enhancement proposals he mentioned)

- [ ] resampling API: scikit-learn/enhancement_proposals#15
- [ ] feature names: scikit-learn/scikit-learn#13307
- [ ] sample properties: scikit-learn/enhancement_proposals#16


Sam followed this by creating an AIF360 branch where he is leading the work, and we ended up annotating it further with what needs to be done internally in AIF360, and externally in sklearn community to make AIF360 datasets, pre-processing, in-processing algorithm compatible with sklearn dataset, transformer and estimator respectively.
https://github.com/IBM/AIF360/tree/sklearn-compat/aif360/sklearn

As part of this, he has identified three additional SLEPs, two in PR, and one accepted.
SLEP007/8 [Feature names] (related to #13307 above)

- [ ] scikit-learn/enhancement_proposals#17
- [ ] scikit-learn/enhancement_proposals#18
- [ ] [SLEP001: Transformers that modify y](https://github.com/scikit-learn/enhancement_proposals/tree/master/slep001/proposal.rst) (workaround viable)

which need to be driven. How much, and what would be needed is we need to dive deeper. In addition he suggests we may need a new SLEP for post-processing algorithms.


**Meeting with Adrin:** @adrinjalali 
Given the approach we are taking is to make AIF360 API compatible with sklearn pipeline APIs, key question then comes where would the code reside. The changes needed in AIF360 to make it work with sklearn pipeline APIs will go in AIF360, and corresponding SLEPs needed to make it happen will be in sklearn.

How will we then expose AIF360 as the default bias checker and mitigator in sklearn community? pip install ai360 package, pip install sklearn package, and then ....currently Sam has

```
from aif360.sklearn.preprocessing import Reweighing
from aif360.sklearn.datasets import fetch_adult
from aif360.sklearn.metrics import disparate_impact_ratio
```

Eventually if all the functionality is duplicated from the original API, we could make the sklearn-compatible API the default and drop .sklearn from the import.

Reverse would make more sense if we want a path from sklearn community outside in, something like

```
from sklearn.algorithms.aif360.preprocessing import Reweighing
from sklearn.metrics.aif360 import disparate_impact_ratio
```

**How can we achieve this by keeping AIF360 independently in its own org, but sklearn apis can be extended to support AIF360?**
user: animeshsingh
issue_14176: state: open
title: Add common test for .transform on different sparse formats
body: As mentioned in https://github.com/scikit-learn/scikit-learn/pull/14140#discussion_r296478787 by @jnothman ,
> `test_truncated_svd.py::test_sparse_formats`
> It's interesting that this is not quite covered by common tests. Apparently, we [check predict and predict_proba](https://github.com/scikit-learn/scikit-learn/blob/1015caf54df347c00152c67adaf1851c5771f0a0/sklearn/utils/estimator_checks.py#L524), but not transform.

It would be useful to extend the `check_estimator_sparse_data` common test to also check `transform` when applicable. The `test_truncated_svd.py::test_sparse_formats` could then be removed. Also possibly other tests (`git grep lil` might give some indications).
user: rth
labels: module:test-suite: everything related to our tests
issue_14175: state: open
title: Refactor BaseEstimator's and Kernel's get_params and set_params
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Hi there, this is a PR for #13555 .

#### What does this implement/fix? Explain your changes.
- Moved `_get_param_names` out of `BaseEstimator` as utility function now called `get_param_names_from_constructor`.
    - This is meant for reuse in `gaussian_process.kernels.Kernel` .
    - Added a `SignatureError` class to pass the signature from the utility function up to the appropriate exception message in `BaseEstimator` and `gaussian_process.kernels.Kernel`.
- Modified `gaussian_process.kernels.Kernel` to use the new utility function and remove duplication.
- Changed `GenericUnivariateSelect._make_selector` to use `get_params` instead of the new utility function so that it raises the proper error in case of a selector with inappropriate signature.
- Also refactored `get_params` to call from a new `_get_params_from` method. This is so `get_params` can be easily overridden to take a signature of, for example, the parent class in custom subclasses of client code.

#### Any other comments?
- Docstrings will be added later.
- Other that raising `SignatureError` (subclassed from `RuntimeError`) these changes don't add or change current behavior, so I haven't added any new tests. If new tests might seem necessary, please let me know.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: alegonz
labels: Needs Decision: Requires decision
issue_14169: state: open
title: Unexpected behavior of canonical correlation analysis (CCA)
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description

Given two view matrices `X` and `Y`, CCA should return the `n_components` linear combinations of the elements in `X`'s and `Y`'s vectors with the largest correlation, i.e the first component has the largest correlation, the second one has the second-largest correlation in a direction that is uncorrelated with the first one, etc. However, in the following code, the second columns are clearly less correlated than the third columns:

#### Steps/Code to Reproduce

```python
from sklearn.cross_decomposition import CCA
import numpy as np
from scipy import stats

X = [[0.006061109337620652, 0.0392466675239141, -0.04312459861840733], [-0.6652995467596429, 0.2076410843346226, -0.7817536882379651], [-1.4060868112838942, -0.055029478343267685, -1.006415484608637], [-2.170613455952169, -0.15770102997315535, -1.5223958036356375], [-2.895702070412092, -0.20498481413822175, -1.8232022285963847], [-3.687452614812402, -0.543003880524402, -2.2952030829468533], [-4.206168972149556, -0.3365693935548624, -2.841946351795423], [-5.231288268781064, -0.8024321344988571, -3.40532581256557], [-6.095805742721522, -0.9381144689340173, -3.593752186094848], [-6.297988264542059, -0.7231985020991631, -3.9435579269998406], [-7.10897027952524, -0.8639925998765747, -4.264992629284153], [-8.116238092376772, -1.0123970020855437, -4.96858178622968], [-8.969468878952105, -1.0235782019578692, -5.617282941713933], [-9.839359511108077, -1.2819621078971968, -5.8901943190245625], [-10.181936322525571, -0.9904671991812529, -6.240811384647836]]
Y = [[0.032927114749911154, 0.21320841666565743, -0.23427536580450153], [1.431742605643286, 0.23963850202268067, 0.8438745303679628], [2.908798834568648, 0.7357229001312737, 1.325345683629048], [4.438824821921929, 0.9473643810538429, 2.35038560647864], [5.887201894166226, 1.0302756424934638, 2.964806513433767], [7.409049064480012, 1.3070946380395154, 4.347473875547982], [8.51501831350366, 1.3380108570442941, 4.9533251686263275], [10.57244384646805, 2.31627294094068, 6.028949244604159], [12.22872203222364, 2.1165257564864675, 6.923464021607424], [12.664660419747504, 1.8911363532121173, 7.398432173930664], [14.29235367239137, 2.2098221962551343, 8.000538342827351], [16.327977920399373, 2.643183255720207, 9.257671785118596], [18.081288169620517, 2.968898443090926, 10.221747267811098], [19.754046559146662, 3.051682253577557, 11.244435627784393], [20.466418131910004, 2.644933083198568, 11.752014917896375]]

cca = CCA(n_components=3)
cca.fit(X, Y)
X_transformed, Y_transformed = cca.transform(X, Y)

print(X_transformed)

[[ 1.64277244  0.08237031  0.11724683]
 [ 1.41457457 -0.11600721  0.07162219]
 [ 1.18686358  0.00666119  0.08822118]
 [ 0.94070119 -0.02504267 -0.0112954 ]
 [ 0.71314666  0.02919558  0.25990473]
 [ 0.46246267  0.05607036 -0.16424275]
 [ 0.28625435 -0.09546609 -0.02850206]
 [-0.03644528 -0.03407977 -0.55790786]
 [-0.30127324  0.12266269 -0.12622283]
 [-0.37581414 -0.01941656 -0.0343278 ]
 [-0.62900674  0.05973748  0.13448604]
 [-0.95400947  0.0082079  -0.05487306]
 [-1.23214839 -0.07548718 -0.08864002]
 [-1.50031366  0.05776429  0.00665019]
 [-1.61776455 -0.0571703   0.38788062]]

print(Y_transformed)

[[ 1.64131294  0.01428169  0.11343087]
 [ 1.41330957 -0.06169376  0.06929115]
 [ 1.18580457  0.04946041  0.0853499 ]
 [ 0.9398609   0.01613582 -0.01092778]
 [ 0.71251056  0.06356946  0.25144578]
 [ 0.46205587 -0.05371303 -0.15889725]
 [ 0.28599564 -0.05298207 -0.02757442]
 [-0.03641563  0.02220818 -0.53975   ]
 [-0.30099795  0.01004611 -0.12211474]
 [-0.37547807 -0.063071   -0.03321056]
 [-0.62844374  0.02679924  0.13010901]
 [-0.95315959  0.0185694  -0.05308714]
 [-1.23105629  0.03726576 -0.08575511]
 [-1.49897395  0.01483407  0.00643375]
 [-1.61632484 -0.04171028  0.37525653]]
```

#### Expected Results
The correlation between the third columns of `X_transfomed` and `Y_transformed` is approximately 1, while the correlation between the second columns is only 0.389. Therefore, I expected the third column to appear in the place of the second column.

#### Actual Results
When I run CCA with `n_components=2` I get the first and second columns, while I should have gotten the first and the third.

#### Versions
Linux-4.13.0-16-generic-x86_64-with-Ubuntu-17.10-artful
Python 3.6.3 (default, Oct  3 2017, 21:45:48) 
NumPy 1.14.3
SciPy 1.2.0
Scikit-Learn 0.19.1

<!-- Thanks for contributing! -->

user: shauli-ravfogel
issue_14145: state: open
title:  LassoLarsIC is missing Xy from its fit method signature which is present in parent parent method.
body: Investigating issue no #12167. Mismatch between signature and use of an overridden method. Source root/sklearn/linear_model/least_angle.py 

The fit function signature  in LassoLarsIC is  not matching the signature of the fit function in the Lars class which its inheriting from.

Should we continue and introduce XY on the method signature on LassoLarsIC on the fit method.
#WiMLDS 

user: Kibugu
labels: Needs Decision - API: Requires decision regarding API
issue_14139: state: open
title: Suggestion: Linear Subspace Classifier
body: Do folks believe that the following classifier is worth implementing?

> Naseem, I., Togneri, R. and Bennamoun, M., 2010. Linear regression for face recognition. IEEE transactions on pattern analysis and machine intelligence, 32(11), pp.2106-2112.

[Link to paper](https://ieeexplore.ieee.org/document/5506092) (~1K citations).

This is a simple classifier that looks at the distance of a test point to the linear subspace spanned by the training examples in each class. Then it predicts the class with the smallest distance. From my understanding, it only works when the number of features is larger than the number of counts for all classes, otherwise the computed linear kernel matrix is singular and therefore not invertible. Posterior probability estimates are not natively supported. 

In the case of high input dimensionality with few samples, however, it might be useful in some cases. From a quick stab I had at implementing the classifier and playing around with toy datasets where `n_features > n_samples`, it seems to be doing a much better job than e.g. `LogisticRegression` or `LinearDiscriminantAnalysis`, which suffers from variable collinearity. 

If the core team think that this is a feature worth implementing, I am happy to submit a PR. 


user: agamemnonc
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_14124: state: open
title: OneVsRestClassifier violates predict(X)==argmax(decision_function)
body: #### Description

When `myclassifier.decision_function` returns only `0`s for some sample, the wrapping OneVsRestClassifier will `predict` the last class instead of the first one.

The cause is in [multiclass.py](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/multiclass.py#L300..L303) where later classes rated 0 by the decision_function override earlier ones.
This violates the documented (and questioned, see #13631) requirement that `the row-wise arg-maximum is the predicted class`.

#### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.multiclass import OneVsRestClassifier
from sklearn.base import BaseEstimator, ClassifierMixin
from sklearn.utils.estimator_checks import check_classifiers_train
from sklearn.utils.validation import check_X_y
```


```python
class Dummy(BaseEstimator, ClassifierMixin):
    def fit(self, X, y):
        X, y = check_X_y(X, y, multi_output=False)
        self.classes_ = np.unique(y)
        self.n_features_ = X.shape[1]
        return self
    def predict(self, X):
        n_features = X.shape[1]
        if self.n_features_ != n_features:
            raise ValueError("Number of features of the model must "
                             "match the input. Model n_features is %s and "
                             "input n_features is %s "
                             % (self.n_features_, n_features))
        return np.full(len(X), self.classes_[0])
    def decision_function(self, X):
        n_features = X.shape[1]
        if self.n_features_ != n_features:
            raise ValueError("Number of features of the model must "
                             "match the input. Model n_features is %s and "
                             "input n_features is %s "
                             % (self.n_features_, n_features))
        return np.zeros(len(X))

class POVR(OneVsRestClassifier):
    def _more_tags(self):
        return {'poor_score': True}
```


```python
>>> x = np.array([[0], [0], [0]])
>>> y = np.arange(3)
>>> d = Dummy().fit(x,y)
>>> d.predict(x)
    array([0, 0, 0])
>>> d.decision_function(x)
    array([0., 0., 0.])
>>> od = POVR(Dummy()).fit(x,y)
>>> od.classes_
    array([0, 1, 2])
>>> od.predict(x)
    array([2, 2, 2])
>>> od.decision_function(x)
    array([[0., 0., 0.],
           [0., 0., 0.],
           [0., 0., 0.]])
>>> np.argmax(od.decision_function(x), axis=1)
    array([0, 0, 0])  # != od.predict(x)

>>> check_classifiers_train('t', POVR(Dummy()))
    ---------------------------------------------------------------------------

    AssertionError                            Traceback (most recent call last)

    <ipython-input-57-e45e548f419d> in <module>
    ----> 1 check_classifiers_train('t', POVR(Dummy()))
    

    /usr/lib/python3.7/site-packages/sklearn/utils/testing.py in wrapper(*args, **kwargs)
        353             with warnings.catch_warnings():
        354                 warnings.simplefilter("ignore", self.category)
    --> 355                 return fn(*args, **kwargs)
        356 
        357         return wrapper


    /usr/lib/python3.7/site-packages/sklearn/utils/estimator_checks.py in check_classifiers_train(name, classifier_orig, readonly_memmap)
       1493                 else:
       1494                     assert_equal(decision.shape, (n_samples, n_classes))
    -> 1495                     assert_array_equal(np.argmax(decision, axis=1), y_pred)
       1496 
       1497                 # raises error on malformed input for decision_function


    /usr/lib/python3.7/site-packages/numpy/testing/_private/utils.py in assert_array_equal(x, y, err_msg, verbose)
        902     __tracebackhide__ = True  # Hide traceback for py.test
        903     assert_array_compare(operator.__eq__, x, y, err_msg=err_msg,
    --> 904                          verbose=verbose, header='Arrays are not equal')
        905 
        906 


    /usr/lib/python3.7/site-packages/numpy/testing/_private/utils.py in assert_array_compare(comparison, x, y, err_msg, verbose, header, precision, equal_nan, equal_inf)
        825                                 verbose=verbose, header=header,
        826                                 names=('x', 'y'), precision=precision)
    --> 827             raise AssertionError(msg)
        828     except ValueError:
        829         import traceback


    AssertionError: 
    Arrays are not equal
    
    Mismatch: 100%
    Max absolute difference: 2
    Max relative difference: 1.
     x: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
     y: array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,...
```

#### Versions
~~~py
/usr/lib/python3.7/site-packages/numpy/distutils/system_info.py:639: UserWarning: 
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  self.calc_info()

System:
    python: 3.7.3 (default, Mar 26 2019, 21:43:19)  [GCC 8.2.1 20181127]
executable: /usr/bin/python3
   machine: Linux-5.1.11-arch1-1-ARCH-x86_64-with-arch

BLAS:
    macros: NO_ATLAS_INFO=1, HAVE_CBLAS=None
  lib_dirs: /usr/lib64
cblas_libs: cblas

Python deps:
       pip: 19.0.3
setuptools: 41.0.1
   sklearn: 0.21.2
     numpy: 1.16.4
     scipy: 1.3.0
    Cython: 0.29.10
    pandas: 0.24.2
~~~
user: azrdev
issue_14104: state: open
title: LabelEncoder accepts continuous y
body: #### Description
When passing a continuous target variable to `LabelEncoder`, I'd expect it to reject instead of encoding its values.

#### Steps/Code to Reproduce

```python
>>> from sklearn.datasets import load_boston
>>> from sklearn.preprocessing import LabelEncoder, LabelBinarizer

>>> b = load_boston()
>>> LabelBinarizer().fit(b.target)
...
ValueError: Unknown label type: (array([24. , 21.6, 34.7, 33.4, 36.2, 28.7, 22.9, ... ]))

>>> LabelEncoder().fit(b.target)
```

#### Expected Results
Also a `ValueError`, maybe with a better message?

#### Actual Results
no error.

~~~py
>>> LabelEncoder().fit(b.target).transform(b.target)                                                                                                           
array([137, 113, 196, 193, 203, 165, 126, 156,  64,   ... ])
~~~

#### Versions

~~~
/usr/lib/python3.7/site-packages/numpy/distutils/system_info.py:639: UserWarning: 
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  self.calc_info()

System:
    python: 3.7.3 (default, Mar 26 2019, 21:43:19)  [GCC 8.2.1 20181127]
executable: /usr/bin/python3
   machine: Linux-5.1.6-arch1-1-ARCH-x86_64-with-arch

BLAS:
    macros: NO_ATLAS_INFO=1, HAVE_CBLAS=None
  lib_dirs: /usr/lib64
cblas_libs: cblas

Python deps:
       pip: 19.0.3
setuptools: 41.0.1
   sklearn: 0.20.3
     numpy: 1.16.4
     scipy: 1.3.0
    Cython: 0.29.10
    pandas: 0.24.2


user: azrdev
labels: Needs Decision: Requires decision
issue_14102: state: open
title: [WIP] Added SelectFromModelCV
body: #### What does this implement/fix? Explain your changes.

This PR implements `SelectFromModelCV`, bringing the equivalent of what `RFECV` is to `RFE`.
This addresses the need of having a cross validation of feature importances for estimators having no intrinsic cross validation like `LassoCV`or eli5's `PermutationImportance`.

#### Any other comments?

I'll add doc and tests if you are interested in continuing with the integration of this feature.
user: H4dr1en
issue_14094: state: open
title: Suggestion: Fill with feature mean instead of 0 for VarianceThreshold.inverse_transform
body: [Today, `sklearn.feature_selection.VarianceThreshold.inverse_transform` method fills in zeros to features which were removed for having too-small variance.
](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.VarianceThreshold.html#sklearn.feature_selection.VarianceThreshold.inverse_transform) This is certainly predictable, easy to implement and easy to explain. 

However, filling in zeros without respect to the data passed to `fit` means that the reconstruction error can become arbitrarily large. For example, suppose that one of the features in your data always takes the value `10**6`. This clearly has zero variance, since it always takes the same value; however, filling in zeros for that feature when the data via `transform` and `inverse_transform` will result in an output which dramatically differs from the input.

Instead, I think it would make sense to fill in the mean of the columns when using `inverse_transform`. The means can be computed and stored from the data passed to `fit`.  This will make the reconstruction of the transformed data via `inverse_transform` more closely reflect the data that was passed to `fit` because any columns which are removed for having variance less than `threshold` must, by definition, be tightly grouped about the sample mean.

Naturally, in the special cases where the sample means of input features passed to `fit` are *already* zero, the proposed `inverse_transform` method will function the same way it does today, as it will fill in zero values for those features.

In terms of code, this just means keeping an array of column means in addition to the indices of the removed columns.

Of course, it's possible that I have missed an important subtlety, or there is a competing concern which outweighs the argument that I've outlined here. If that's the case, I'd like to know what I've missed!
user: Sycor4x
labels: Needs Decision: Requires decision
issue_14082: state: open
title: Add a test to ensure that estimators do not have invalid estimator tags
body: From #14043, maybe we can add a test to ensure that there's no invalid estimator tags, but we need to make sure that other projects are easy to define their own tags.
user: qinhanmin2014
issue_14063: state: open
title: Programatically finding all supported solvers and losses for an estimator
body: It would be useful to have some mechanism of determining supported solvers for a given estimator. First, because currently `check_estimator` only runs on the default solver and so we are potentially not testing a number of configurations.

It would also make easier to check that all solvers yield comparable results such as https://github.com/scikit-learn/scikit-learn/pull/13914

Of course, this can also be generalized to other parameters that impact the solver but should not change the way estimator behaves.

One way could be just to store e.g. `_supported_solvers` as a private attribute, or possibly in type annotations (though this would require [PEP 586](https://www.python.org/dev/peps/pep-0586/) from Python 3.8 that can be backported by vendoring [`typing_extensions.py`](https://github.com/python/typing/blob/master/typing_extensions/src_py3/typing_extensions.py)), related to #11170 

Another way could be to have some method that yields possible estimator variants to be tested. @amueller if I remember correctly you mentioned something similar in the estimator tags PR.
user: rth
issue_14057: state: open
title: All tests in `check_estimator` are disabled if `X_types` does not include `"2darray"`
body: #### Description
All tests in in `check_estimator` are disabled if `X_types` does not include `"2darray"`. Some tests such as `check_get_params_invariance` or `check_set_params` do not require a `fit` step, and should be run no matter what.

I am currently building a library that uses its own data types, but tries to be Scikit-learn compatible. Thus, I try to make estimators with the same API as Scikit-learn, but different input and output types. It would be useful to check API conformance using `check_estimator` with our estimator objects.

user: vnmabus
issue_14054: state: open
title: var_smoothing parameter of GaussianNB isn't documented nor tested
body: `var_smoothing` was introduced in https://github.com/scikit-learn/scikit-learn/pull/9681

It's quite unclear from the doc why one would want to use this parameter, and what it does. A short section in the user guide would be helpful.

Same for the `epsilon_` attribute that is attached to it.

It would also need some tests, and some input validation (I assume it should be in [0, 1], or at least > 0).
user: NicolasHug
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_14042: state: open
title: RFC design of random_state
body: This is inspired by #14034 but we had several similar issues of the years.
The current design of ``random_state`` is often hard to understand and confusing for users.
I think we should rethink how ``random_state`` works. Maybe that's a 1.0 issue, I'm not sure.

What I find most confusing is the behavior of passing a RandomState object to ``random_state``, because that makes the object be statefull across calls to ``fit``, so it violates our contract of ``fit`` being idempotent. Because ``fit`` (or possibly even ``predict``?) consumes the random state object it's mutated.
I don't see a real use-case for passing a random state object and think we might want to deprecate that. At least we should never store it imho.

There have been countless bugs because of this, and I think they are pretty easily avoidable.

Another question is the behavior of ``random_state=None`` which can also be confusing. Repeated calls to ``fit`` result in different models. Sometimes that's good, sometimes that's bad. There would be ways to change this, but I'm not sure if it's a good idea.
The bug in #14034 is also present when ``random_state=None``, so just deprecating passing ``RandomState`` would not avoid these kinds of bugs in the future.
user: amueller
issue_14039: state: open
title: TREE: warm_start with ccp_alpha
body: Related to https://github.com/scikit-learn/scikit-learn/pull/12887#issuecomment-499559966

#12887 mutates the `self.tree_` of the tree estimator, and hence if we want a `warm_start` on `ccp_alpha`, it should be noted that it has a direction (`ccp_alpha` can only be increased), and for the grid search to know that(?). Otherwise, the estimator should ignore the warm_start and fit a new one.

It may be related to the `warm_start_on` or other related info we were planning to have as a part of the estimators. Is this a good candidate for the estimator tags?
user: adrinjalali
labels: Needs Decision: Requires decision
issue_14038: state: open
title: TREE: public function/method to return a new estimator with a new ccp_alpha 
body: related: https://github.com/scikit-learn/scikit-learn/pull/12887#issuecomment-499559966

After #12887, we should decide if we want a public function/method which would return a tree estimator, pruned according to a new `ccp_alpha`.

To me it makes sense to have a method which doesn't change `self`, and can be used like:

    pruned_tree = orig_tree.prune(ccp_alpha=new_alpha)

Also, do we want to expose the same method on the tree based ensemble models?
user: adrinjalali
labels: Needs Decision: Requires decision
issue_14037: state: open
title: `prange` in trees
body: related to #13213 and triggered by https://github.com/scikit-learn/scikit-learn/pull/12887#discussion_r291233707

we train trees in parallel in tree ensembles (IIRC), so I'm not sure how much we'd benefit from using `prange` in the tree code base. Still, some benchmarks would be nice before we decide how to proceed.

The benchmark would compare the master, with a version of trees where some of the for loops on `range` are done with `prange` instead, and maybe run the ensembles with only one job.
user: adrinjalali
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
Hard: Hard level of difficulty
issue_14036: state: open
title: [QST] Updating indices creation for split
body: I am interested in using GridSearchCV machinery from sklearn with non-numpy libraries like [cuDF](https://github.com/rapidsai/cudf) or [dask.array](http://docs.dask.org/en/latest/array.html).  In exploring this work, I think functions like `_iter_test_indices` and `_iter_test_masks` could be amended slightly to return slices instead of numpy arrays.

For example in `_iter_test_indices`:
https://github.com/scikit-learn/scikit-learn/blob/ec35ed226ca104e841238f2fac24269c2f9f2730/sklearn/model_selection/_split.py#L423-L436

If there is a shuffle, much of the code could be encapsulated in the shuffle conditional.  And when there is
no shuffle, we can yield the appropriate slice object: `slice(0, 71)`...

I wanted to check here before proceeding with the work and/or if folks have any advice before I begin?
user: quasiben
issue_14034: state: open
title: data leak in GBDT due to warm start
body: (This is about the non-histogram-based version of GBDTs)

X is split into train and validation data with `train_test_split(random_state=self.random_state)`.

As @johannfaouzi noted, in a warm starting context, this will produce a leak if If `self.random_state` is a `RandomState` instance: some samples that were used for training in a previous `fit` might be used for validation now.

~~I think the right fix would be to raise a `ValueError` if the provided random state isn't a number and early-stopping is activated~~
user: NicolasHug
issue_14011: state: open
title: Add Matrix Profile algorithm to scikit-learn
body: Matrix Profile is an algorithm developed by the Keogh research group at UC-Riverside (https://www.cs.ucr.edu/~eamonn/MatrixProfile.html) that serves as an extremely powerful tool for finding patterns, anomalies and evolving behaviors in massive timeseries datasets. We released a Python version of Matrix Profile last year (matrixprofile-ts) and it seems like a good idea to align further development with Python machine learning best practices, as well as to broaden the user/developer base. 
user: vanbenschoten
labels: Needs Decision - Close: Requires decision for closing
issue_14001: state: open
title: Run itrees in parallel during prediction.
body: #### Reference Issues/PRs
https://github.com/scikit-learn/scikit-learn/issues/14000

#### What does this implement/fix? Explain your changes.
Isolation Forest is executed in parallel during fitting. But, during prediction it is running single-threaded.

In this PR, I parallelised the execution during prediction, more precisely in the _compute_score_samples method. Each ITree was being called in sequence, using a for loop. I created an auxiliary internal function that executes each tree, and this function can be run in parallel. I used joblib for parallelisation.

#### Any other comments?
I run the tests for Isolation Forest and it passed.


user: sergiormpereira
labels: Needs Decision: Requires decision
issue_14000: state: open
title: Isolation Forest executes single-threaded during prediction
body: #### Description
Isolation Forest is executed in parallel during fitting. But, during prediction it is running single-threaded. This translates into being quite slow at prediction time.

Together with this issue I am submitting a PR that parallelises prediction.

#### Steps/Code to Reproduce
NA

#### Expected Results
NA
#### Actual Results
NA

#### Versions
python: 3.6.8
sklearn: 0.22.dev0
numpy: 1.16.4
scipy: 1.3.0
Cython: 0.29.9
user: sergiormpereira
issue_13998: state: open
title: `RidgeCV` `cv_values_` are for preprocessed data: centered and scaled by sample weights.
body: when `store_cv_values=True`, `RidgeCV` stores the leave-one-out squared errors,
when `scoring=None`, or the leave-one-out predictions, when `scoring` is
provided by the user, in its `cv_values_` attribute.

However, when `scoring` is not `None`, it stores the predictions for the
preprocessed data, i.e. rescaled by the square roots of the sample weights and
with the mean of `y` removed:

```python
import numpy as np
from sklearn.linear_model import RidgeCV
from sklearn.datasets import make_regression

x, y = make_regression(n_samples=6, n_features=2, random_state=0)
squared_error = RidgeCV(
    store_cv_values=True, alphas=[10.]).fit(x, y).cv_values_.ravel()
custom_scoring = RidgeCV(
    store_cv_values=True, scoring='neg_mean_squared_error',
    alphas=[10.]).fit(x, y)
# to get the actual predictions we need to add the y mean
custom = (y - (custom_scoring.cv_values_.ravel() + y.mean()))**2
assert np.allclose(squared_error, custom)

sw = np.arange(6) + 1
squared_error = RidgeCV(store_cv_values=True, alphas=[10.]).fit(
    x, y, sample_weight=sw).cv_values_.ravel()
custom_scoring = RidgeCV(
    store_cv_values=True, scoring='neg_mean_squared_error',
    alphas=[10.]).fit(x, y, sample_weight=sw)
# to get the actual predictions we need to rescale by inverse square root
# sample weights and add the y mean
custom = sw * (y
               - (custom_scoring.cv_values_.ravel() / np.sqrt(sw)
                  + np.average(y, weights=sw)))**2
assert np.allclose(squared_error, custom)
```

I think that for a user, it would be easier to get directly the predictions in
the original space, and not need to do this post-processing of `cv_values_`.

Should we rescale the cv values and add the intercept during `fit`?
user: jeromedockes
labels: Needs Decision: Requires decision
issue_13994: state: open
title: Kernel Linear Discriminant Analysis 
body: Hi all, 

My group is interested in implementing Kernel Linear Discriminant Analysis (LDA) with the paper described here http://www.kernel-machines.org/papers/upload_21840_GDA.pdf and I'd like to know if this is a worthwhile pursuit, whether it's already been done before, and if not what pointers you have.

It fulfills the algorithms requirements 
- at least 3 years since publication (year 2000) 
- 200+ citations (1965) 
- and wide use and usefulness. 

Additionally, PCA happens to have a kernel PCA method with various kernels so I believe it is a valuable addition. 

I also had some additional questions: 

What is the exact difference between https://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html and https://scikit-learn.org/0.16/modules/generated/sklearn.lda.LDA.html ? They look to be exactly the same. 
**I now understand this commit migrated it for naming https://github.com/scikit-learn/scikit-learn/commit/679936f6bcb66b1b734e4f3609bdb203e41d8460.**

What is the exact variant of Linear Discriminant Analysis already implemented? It looks to be https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.10.3633 which is class-dependent LDA from a mixture of gaussians but I'm not sure and would like to confirm that.
**Looks like I was mistaken and the classifier uses Bayes rule, I believe the dimensionality reduction is the same.**

Going to @ some people involved from git blame, hope you don't mind. 
@cbrnr @mblondel @vinayak-mehta @bobchennan @agramfort @amueller 

user: k--chow
issue_13989: state: open
title: GaussianProcessRegressor multi-output underlying mechanics?
body: GaussianProcessRegressor allows multi-dimensional outputs.

However, the docs is lacking in explaining how these multi-outputs are predicted. 

Does the model assume correlation between these multi-outputs, as in 'co-kriging' or multi-task or multi-output Gaussian Processes techniques?

user: bread9
issue_13988: state: open
title: Avoid memory copies in linear_model when fit_intercept=False
body: Partially addresses #13986 and #13923 

This avoids one memory copy in linear models when `copy_X=True` (default) and `fit_intercept=False`.

For instance, this makes `Ridge(fit_intercept=False)` around 10% faster on the following dataset,
```py
from sklearn.datasets import make_blobs
  
X, y = make_blobs(n_samples=100000, n_features=10, random_state=42)
```

Maybe we should change the default value of `copy_X` from `True` to `None` but I'm not sure it's worth a deprecation cycle.

I have checked that this doesn't break anything with the following common test,
<details>

```diff
diff --git a/sklearn/utils/estimator_checks.py b/sklearn/utils/estimator_checks.py
index 3b09ec287..72961bd73 100644
--- a/sklearn/utils/estimator_checks.py
+++ b/sklearn/utils/estimator_checks.py
@@ -124,6 +124,11 @@ def _yield_classifier_checks(name, classifier):
     # basic consistency testing
     yield check_classifiers_train
     yield partial(check_classifiers_train, readonly_memmap=True)
+
+    sig = signature(classifier.__class__.__init__)
+    if "copy_X" in sig.parameters:
+        yield check_extra_copy_fit_intercept
+
     yield check_classifiers_regression_target
     if not tags["no_validation"]:
         yield check_supervised_y_no_nan
@@ -171,6 +176,11 @@ def _yield_regressor_checks(name, regressor):
     yield check_regressor_data_not_an_array
     yield check_estimators_partial_fit_n_features
     yield check_regressors_no_decision_function
+
+    sig = signature(regressor.__class__.__init__)
+    if "copy_X" in sig.parameters:
+        yield check_extra_copy_fit_intercept
+
     if not tags["no_validation"]:
         yield check_supervised_y_2d
     yield check_supervised_y_no_nan
@@ -190,6 +200,7 @@ def _yield_transformer_checks(name, transformer):
     yield check_transformer_general
     yield partial(check_transformer_general, readonly_memmap=True)
 
+
     if not _safe_tags(transformer, "stateless"):
         yield check_transformers_unfitted
     # Dependent on external solvers and hence accessing the iter
@@ -956,6 +967,25 @@ def check_transformer_general(name, transformer, readonly_memmap=False):
     _check_transformer(name, transformer, X.tolist(), y.tolist())
 
 
+@ignore_warnings(category=(DeprecationWarning, FutureWarning))
+def check_extra_copy_fit_intercept(name, transformer):
+    """Check linear models work on RO arrays when fit_transform=False"""
+
+    X, y = make_blobs(n_samples=30, centers=[[0, 0, 0], [1, 1, 1]],
+                      random_state=0, n_features=2, cluster_std=0.1)
+    X = StandardScaler().fit_transform(X)
+    X -= X.min()
+
+    X, y = create_memmap_backed_data([X, y])
+
+    estimator = clone(transformer)
+    estimator.set_params(copy_X=False)
+    sig = signature(estimator.__class__.__init__)
+    if "fit_intercept" in sig.parameters:
+        estimator.set_params(fit_intercept=False)
+    estimator.fit(X, y)
+
+
 @ignore_warnings(category=(DeprecationWarning, FutureWarning))
 def check_transformer_data_not_an_array(name, transformer):
     X, y = make_blobs(n_samples=30, centers=[[0, 0, 0], [1, 1, 1]],
```
</details>

but of course this only checks the default solver and parameters (and we don't yet have a way to programmatically check all solvers I think?). I'm not sure it's worth adding it in this PR.
user: rth
issue_13986: state: open
title: Change default copy value from True to None
body: A fair amount of estimators currently have `copy=True` (or `copy_X=True`) by default. In practice, this means that the code looks something like,
```py
X = check_array(X, copy=copy)
```
and then some other calculations that may change or not X inplace. In the case when the following operations are not done inplace, we have just made a wasteful copy with no good reason.

As discussed in https://github.com/scikit-learn/scikit-learn/issues/13923, an example is for instance `Ridge(fit_intercept=False)` that will copy X, although it is not needed. ~~Actually, I can't find any inplace operations of `X` in `Ridge` even with `fit_intercept=True`, but maybe I am missing something.~~ (found it)

I think in general it would be better to avoid the,
```py
X = check_array(X, copy=copy)
```
pattern, and instead make a copy explicitly where it is needed. Maybe it could  be OK to not make a copy with `copy=True` if no copy is needed. Alternatively we could introduce `copy=None` by default.

Adding a common test that checks that `Estimator(copy=True).fit(X, y)` doesn't change `X`.
user: rth
labels: Needs Decision: Requires decision
issue_13972: state: open
title: how can cv folds can be more than number of groups in cross_validate?
body: In the following code:
```
from sklearn import datasets, linear_model
from sklearn.model_selection import cross_validate
from sklearn.metrics.scorer import make_scorer
from sklearn.metrics import confusion_matrix
from sklearn.svm import LinearSVC
import numpy as np

digits = datasets.load_digits()

X = digits.data[:500]
y = digits.target[:500]

counts, unique = np.unique(y, return_counts=True)

g1 = np.repeat(0, int(len(y)/2))
g2 = np.repeat(1, int(len(y)/2))

g = np.concatenate((g1, g2))

linear = LinearSVC()

cv_results = cross_validate(linear, X, y, cv=5)
sorted(cv_results.keys())
print(cv_results)


print('cv_results')
print(cv_results['test_score'])


print('cross-validate')
scores = cross_validate(linear, X, y, cv=5,
                        scoring=('f1_weighted'),
                        return_train_score=True, groups=g)
print(scores)

```

Shouldn't it be returned that:
groups should be equal to CV for the folds?


user: omarcr
issue_13969: state: open
title: Add non-strict mode to check_estimator
body: Right now the common tests have two different use-cases: checking internal sklearn estimators, and checking 3rd party estimators via ``check_estimator``.

The common tests are very strict to a degree that's not really reasonable for 3rd party estimators, in particular in checking for particular error messages.
Furthermore, the required error-messages are not documented anywhere (see also #10082).

I think it would be good to add a ``strict=False`` argument to check_estimator to not require exact messages, and possibly not even the exact error types. That would make it much easier for people to use ``check_estimator``.

user: amueller
issue_13962: state: open
title: Use of target_names in example datasets is inconsistent
body: #### Description

It seems `target_names` sometimes correspond to values of categorical target and sometimes correspond to target colum name itself.

#### Steps/Code to Reproduce

```python
from sklearn.datasets import load_linnerud
from sklearn.datasets import load_iris

print(load_linnerud()['target_names'])
print(load_iris()['target_names'])
```

#### Expected Results

I would expect consistency. Or that `target_names` target column names (even in the case of one column), or that it returns categorical target values (and then it should not exist for regression).

#### Actual Results

```
['Weight', 'Waist', 'Pulse']
array(['setosa', 'versicolor', 'virginica'], dtype='<U10')
```

Also note that once it is a list and once it is an array.

#### Versions

```
System:
    python: 3.6.7 (default, Oct 22 2018, 11:32:17)  [GCC 8.2.0]
executable: /usr/bin/python3
   machine: Linux-4.18.0-20-generic-x86_64-with-Ubuntu-18.04-bionic

BLAS:
    macros: HAVE_CBLAS=None
  lib_dirs: /usr/lib/x86_64-linux-gnu
cblas_libs: openblas, openblas

Python deps:
       pip: 19.0.3
setuptools: 41.0.1
   sklearn: 0.20.3
     numpy: 1.15.4
     scipy: 1.2.1
    Cython: None
    pandas: 0.23.4
```
user: mitar
issue_13942: state: open
title: [MRG] Adds Bernoulli mixture model to the `mixture` module
body: #### Reference Issues/PRs

Fixes #12055

#### What does this implement/fix? Explain your changes.

This implements a Bernoulli mixture model. It reuses much of the code in the 
`GaussianMixture` class, mainly differing in the log-likelihood calculation.

#### Any other comments?

An example of applying this to images is provided. The example is motivated 
by the discussion in Chapter 9 of *Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning*.
user: ben-dilday-enigma
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_13935: state: open
title: [MRG] decision_path and apply for all tree-based ensemble model classes.
body: #### What does this implement/fix? Explain your changes.
The `decision_path` method is currently available only for `RandomForest`/`ExtraTrees` Classifier/Regressor, but not for `IsolationForest` and `GradientBoosting` Classifier/Regressor. 
The `apply` method is currently available only for `RandomForest`/`ExtraTrees`/`Gradient Boosting` Classifier/Regressor but not for `IsolationForest` or the `Histogram`-variants of gradient boosting.

I have added a mixin, `ForestMixin`, in `sklearn.ensemble.base` to provide a common implementation of the methods `decision_path` and `apply` for `RandomForest`/`ExtraTrees` Classifier/Regressor and `IsolationForest`.
However, `Gradient Boosting` Classifier/Regressor require specific implementations of the methods `decision_path` and `apply`. The `apply` method is already available. Work in progress for `decision_path` ...
user: ab-anssi
labels: Needs Decision: Requires decision
issue_13923: state: open
title: Linear regression unnecessarily slow
body: TLDR so far: sklearn uses SVD which is more numerically stable, but perhaps it should default to something quicker such as suggested below unless SVD is needed. Later post contains detailed suggestion.

#### Description
Standard linear regression implementation seems unnecessarily slow, at least in many cases. May extend to other linear estimators e.g. Ridge.

#### Steps/Code to Reproduce
```py
from sklearn.linear_model import LinearRegression
import numpy as np
import time

def mylin(X,Y):
    CXX = np.dot(X.T,X)/X.shape[0]
    CXY = np.dot(X.T,Y)/X.shape[0]
    return np.linalg.solve(CXX,CXY).T
    
X = np.random.normal(size=[500000,40])
Y = np.random.normal(size=[500000,10])

lin = LinearRegression(False)

t = time.time()
lin.fit(X,Y)
print('Time to fit sklearn linear:', time.time()-t)

t = time.time()
coef = mylin(X,Y)
print('Time to fit analytic linear regression solution:', time.time()-t)

print('Correlation between solutions:', np.corrcoef(coef.flatten(), lin.coef_.flatten())[0,1])
```

#### Results
On my laptop, I see the following results:
```
Time to fit sklearn linear: 0.5804157257080078
Time to fit analytic linear regression solution: 0.023940324783325195
Correlation between solutions: 1.0
```

I see a similar speedup on my work machine, which is also a conda install but on Ubuntu 18.04

#### Versions

> Could not locate executable g77
> Could not locate executable f77
> Could not locate executable ifort
> Could not locate executable ifl
> Could not locate executable f90
> Could not locate executable efl
> Could not locate executable gfortran
> Could not locate executable f95
> Could not locate executable g95
> Could not locate executable efort
> Could not locate executable efc
> Could not locate executable flang
> don't know how to compile Fortran code on platform 'nt'
> 
> System:
>     python: 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)]
> executable: C:\Users\---\Anaconda3\pythonw.exe
>    machine: Windows-10-10.0.17763-SP0
> 
> BLAS:
>     macros: 
>   lib_dirs: 
> cblas_libs: cblas
> 
> Python deps:
>        pip: 18.1
> setuptools: 40.6.3
>    sklearn: 0.20.1
>      numpy: 1.16.3
>      scipy: 1.1.0
>     Cython: 0.29.2
>     pandas: 0.23.4
> C:\Users\ShakesBeer\Anaconda3\lib\site-packages\numpy\distutils\system_info.py:638: UserWarning: 
>     Atlas (http://math-atlas.sourceforge.net/) libraries not found.
>     Directories to search for the libraries can be specified in the
>     numpy/distutils/site.cfg file (section [atlas]) or by setting
>     the ATLAS environment variable.
>   self.calc_info()
> C:\Users\ShakesBeer\Anaconda3\lib\site-packages\numpy\distutils\system_info.py:638: UserWarning: 
>     Blas (http://www.netlib.org/blas/) libraries not found.
>     Directories to search for the libraries can be specified in the
>     numpy/distutils/site.cfg file (section [blas]) or by setting
>     the BLAS environment variable.
>   self.calc_info()
> C:\Users\ShakesBeer\Anaconda3\lib\site-packages\numpy\distutils\system_info.py:638: UserWarning: 
>     Blas (http://www.netlib.org/blas/) sources not found.
>     Directories to search for the sources can be specified in the
>     numpy/distutils/site.cfg file (section [blas_src]) or by setting
>     the BLAS_SRC environment variable.
>   self.calc_info()

Worth mentioning that I am using MKL, and `np.show_config()` confirms this.
user: ShakesB33r
issue_13921: state: open
title: Error when computing the gradient of a derivative with Gaussian process regression in scikit
body: Gaussian process regression in scikit-learn provides a conditional mean and variance, (y_*,\sigma_*^2), based upon the observed data, (y,\sigma^2). But given (y_*,\sigma_*^2), how can you compute the variance of \frac{dy_*}{dx_*} in scikit-learn? The formula for doing this is given by [equation (9.1)](http://www.gaussianprocess.org/gpml/chapters/RW9.pdf), although when applying this method in the below code, I receive erroneous results. Here is a working example:  

    import numpy as np
    from matplotlib import pyplot as plt
    from sklearn.gaussian_process import GaussianProcessRegressor
    from sklearn.gaussian_process.kernels import RBF, ConstantKernel as C
    def f(x):
        """The function to predict."""
        return 1.5*(1. - np.tanh(100.*(x-0.96))) + 1.5*x*(x-0.95) + 0.4 + 1.5*(1.-x)* np.random.random(x.shape)
    
    X = np.array([0.803,0.81,0.82,0.827,0.83,0.84,0.845,0.861,0.875,0.892,0.905,
                    0.91,0.92,0.925,0.935,0.941,0.947,0.96,
                    0.974,0.985,0.995,1.0])
    X = np.atleast_2d(X).T
    
    # Observations and noise
    y = f(X).ravel() 
    noise = 0.6/y
    y += noise
     
    kernel = C(1.0, (1e-10, 1000)) * RBF(100, (1e-4, 1e4))
    gp = GaussianProcessRegressor(kernel=kernel, alpha=noise ** 2,
                                  n_restarts_optimizer=50)
    
    gp.fit(X, y)
     
    x = np.atleast_2d(np.linspace(0.8, 1.02, 1000)).T
    y_pred, sigma = gp.predict(x, return_std=True)
    
    plt.figure() 
    plt.errorbar(X.ravel(), y, noise, fmt='k.', markersize=10, label=u'Observations')
    plt.plot(x, y_pred, 'k-', label=u'Prediction')
    plt.fill(np.concatenate([x, x[::-1]]),
             np.concatenate([y_pred - 1.9600 * sigma,
                            (y_pred + 1.9600 * sigma)[::-1]]),
             alpha=.1, fc='k', ec='None', label='95% confidence interval')
    plt.xlabel('x')
    plt.ylabel('y')
    plt.xlim(0.8, 1.02)
    plt.ylim(0, 5)
    plt.legend(loc='lower left')
    plt.show()
    
provides me with the expected result for (y_*,\sigma_*^2):

[![y][1]][1]

But now I would like to compute \frac{dy_*}{dx_*} and its associated error. I try this via 2 approaches:

**1)** I know that k(x_*,x_*) = \sigma^2_*, so if I simply take the gradient of \sigma^2_* with respect to x_* twice, then I should recover \frac{d^2k(x_*,x_*)}{dx_*^2}, which is exactly equal to the expression in (9.1). When I do this: 

    dkdx = np.gradient(sigma**2.,x[:,0])
    d2kdx2 = np.gradient(dkdx,x[:,0]) 
    dydx = np.gradient(y_pred,x[:,0])
    d2kdx2_old = d2kdx2*1.
    d2kdx2[d2kdx2 < 0.] = 0.01
    inds_new = np.where(d2kdx2 >= 0.) #I set all negative errors to 0 for plotting purposes
    cov_grad_used = d2kdx2[inds_new]
    x_plot = x[inds_new]
    dydx_err_plot = np.sqrt(cov_grad_used)
    dydx_plot = dydx[inds_new]
    
    plt.figure()  
    plt.plot(x_plot, dydx_plot, 'k-', label=u'Prediction')
    plt.fill(np.concatenate([x_plot, x_plot[::-1]]),
             np.concatenate([dydx_plot - 1.9600 * dydx_err_plot,
                            (dydx_plot + 1.9600 * dydx_err_plot)[::-1]]),
             alpha=.1, fc='k', ec='None', label='95% confidence interval')
    plt.xlabel('x')
    plt.ylabel('dy/dx (using sigma)')
    plt.legend(loc='lower left')
    plt.show()

[![dy/dx v1][2]][2]

the output looks almost right, but when I compute `np.min(d2kdx2)`, or, equivalently, `np.gradient(dkdx,x[:,0])`, I get a negative quantity! Yet we know that the variance of \frac{dy_*}{dx_*} should be purely positive! 

**2)** When I try an alternative but supposedly equivalent approach of evaluating the trained covariance kernel directly via `gp.kernel_` and trying to take the second derivative of the values along the diagonal, I get simply zero. This is displayed below:
    
    i = 0
    just_k = []
    while i < len(x):
        kern_eval = gp.kernel_(np.array([x[i][0]]).reshape(1,-1),np.array([x[i][0]]).reshape(1,-1))[0][0]
        just_k.append(kern_eval)
        i = i + 1
        
    just_k = np.array(just_k)
    K_used = just_k
    dkdx_kern = np.gradient(K_used,x[:,0]) #derivative of kernel (along diagonal) evaluated for the radial profile
    d2kdx2_kern = np.gradient(dkdx_kern,x[:,0]) #second derivative, which would correspond to error on dy/dx
    
    inds_new_kern = np.where(d2kdx2_kern >= 0.)
    cov_grad_used_kern = d2kdx2_kern[inds_new_kern]
    x_plot_kern = x[inds_new_kern]
    dydx_err_plot_kern = np.sqrt(cov_grad_used_kern)
    dydx_plot_kern = dydx[inds_new_kern]
    
    plt.figure()  
    plt.plot(x_plot_kern, dydx_plot_kern, 'k-', label=u'Prediction')
    plt.fill(np.concatenate([x_plot_kern, x_plot_kern[::-1]]),
             np.concatenate([dydx_plot_kern - 1.9600 * dydx_err_plot_kern,
                            (dydx_plot_kern + 1.9600 * dydx_err_plot_kern)[::-1]]),
             alpha=.1, fc='k', ec='None', label='95% confidence interval')
    plt.xlabel('x')
    plt.ylabel('dy/dx (using kernel)')
    plt.legend(loc='lower left')
    plt.show()

[![enter image description here][3]][3]

where `np.min(np.gradient(dkdx_kern,x[:,0]))` and `np.max(np.gradient(dkdx_kern,x[:,0]))` both equal zero; or equivalently, `d2kdx2_kern` is zero everywhere! This makes sense mathematically since I am applying a stationary kernel, but given (9.1), I thought this quantity was supposed to represent the variance of my derivative, so something appears off here.

Essentially these are two examples of codes trying to compute the variance associated to the derivative of y_*. But why do I get negative values for the derivative's variance using the first approach (i.e. via the second derivative of the conditional variance), and only zero using the second approach (i.e. via the second derivative of `gp.kernel_`)? What step(s) are necessary to compute the variance of \frac{dy_*}{dx_*} properly using scikit-learn's Gaussian process framework?

  [1]: https://i.stack.imgur.com/lkLHa.png
  [2]: https://i.stack.imgur.com/L0UGr.png
  [3]: https://i.stack.imgur.com/AtNWr.png
user: AbhilashMathews
issue_13917: state: open
title: Feature Request: DummyRegressor Per Category
body: The current [DummyRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyRegressor.html) supports mean, median etc. for the whole dataset. What would be a very convenient addition is, if there was an argument that the DummyRegressor uses for grouping. Then it could predict the mean, median etc. not for the whole dataset, but per grouping variable. I often encounter the situation where I want to build a model per some group or category and this would really help.
user: r0f1
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_13908: state: open
title: [MRG] Fix formula of the objective function in BayesianRidge
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fix #13852


#### What does this implement/fix? Explain your changes.
I fixed the constant terms of the objective function in BayesianRidge.
In addition, the timing of convergence was changed to make the objective function value ​​and the result ``coef_`` consistent.

#### Any other comments?
- Also the objective function of ARDRegression ~~can be~~ was fixed.
- ~~The description of ``scores_`` and the name of the method may be altered as discussed in #13852 .~~
- This fix may conflict with PR #13618, but can be easily resolved.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: c56pony
issue_13852: state: open
title: BayesianRidge scores_ formula seems a bit inaccurate
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
In ``_log_marginal_likelihood`` of BayesianRidge, some of the constant terms seem to be omitted.
https://github.com/scikit-learn/scikit-learn/blob/17a253b0c748cd2ea0f64f8aa2911daf129ddff8/sklearn/linear_model/bayes.py#L351-L358
This code only calculates ![texclip20190510122056](https://user-images.githubusercontent.com/40843206/57500366-1c265c80-731e-11e9-86bb-1fc5989495d7.png) of the constant terms, which is a bit incorrect and confusing.

This problem is not really important, but I think it would be better to fix it.
Please let us know what you think.

#### Exact expressions
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
The reference (Tipping, 2001) contains equations for log marginal likelihood (eq. 36, 38 and 40),
but there is no expression of constant terms.

So, I derived the formula as follows, 
![image](https://user-images.githubusercontent.com/40843206/57502732-fe5df500-7327-11e9-8901-06c0e26d34e9.png)
where N is ``n_samples``, M is ``n_features`` and mu is ``coef``. (The red terms are currently omitted.)
user: c56pony
issue_13846: state: open
title: Add Hypothesis Testing
body: It may be beneficial to add hypothesis testing to some of our tests. Hypothesis testing allows for us to verify mathematical properties such as commutativity: 

```python
from hypothesis import given
import hypothesis.strategies as st

@given(st.integers(), st.integers())
def test_ints_are_commutative(x, y):
    assert x + y == y + x
```

This allows the CI to look for edge cases for us. This would be useful testing metrics. https://hypothesis.readthedocs.io/en/latest/
user: thomasjpfan
labels: Hard: Hard level of difficulty
module:test-suite: everything related to our tests
issue_13794: state: open
title: train_test_split with stratify causing integer overflow (32bit version)
body: Description:
Train_test_split with stratify is returning faulty results due to integer overflow (32bit version only).

Steps to reproduce:
```
import pandas as pd
from sklearn.model_selection import train_test_split
import numpy as np

df = pd.DataFrame(data={'a': np.random.rand(100000), 'b': 0})
df.loc[np.random.randint(0, 100000, 1000), 'b'] = 1
tr, ts = train_test_split(df, test_size=.2, stratify=df['b'])
print(tr.shape, ts.shape)
```

Actual result:
`(93105, 2) (38, 2)`

Expected result:
`(80000, 2) (20000, 2)`

Versions:
Python 3.7.0 (32bit)
Sklearn 0.20.3 
Pandas 0.23.4

user: tkoller78
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_13781: state: open
title: ColumnTransformer doesn't accept some valid Pandas column names
body: #### Description

Pandas columns can be tuples, or in the case of a MultiIndex, can be accessed using a tuple. `ColumnTransformer` does not support this. It seems to be an oversight in the data validation, specifically in how [`_check_key_type()`](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/compose/_column_transformer.py#L560-L588) is used.

#### Steps/Code to Reproduce

```python
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler

columns_index = pd.MultiIndex.from_tuples([('a', 1), ('a', 2), ('b', 0)])

d1 = pd.DataFrame([
    [1, 2, 3],
    [4, 5, 6]
], columns=columns_index)

ct = ColumnTransformer([
    ('scaler', StandardScaler(), [('a', 1), ('a', 2)])
])

d2 = ct.fit_transform(d1)
```

#### Expected Results

`ColumnTransformer` accesses columns `('a', 1)` and `('a', 2)`, and sends them to the `StandardScaler` instance.

#### Actual Results

```
ValueError: No valid specification of the columns. Only a scalar, list or slice of all integers or all strings, or boolean mask is allowed
```

#### Versions

```
System:
    python: 3.7.3 (default, Mar 27 2019, 22:11:17)  [GCC 7.3.0]
executable: /<redacted>/conda-env/bin/python
   machine: Linux-3.10.0-514.26.2.el7.x86_64-x86_64-with-redhat-7.3-Maipo

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: /<redacted>/lib
cblas_libs: mkl_rt, pthread

Python deps:
       pip: 19.0.3
setuptools: 40.8.0
   sklearn: 0.20.3
     numpy: 1.16.2
     scipy: 1.2.1
    Cython: 0.29.6
    pandas: 0.24.2
```
user: gwerbin
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_13770: state: open
title: repr of column transformer unhelpful
body: Here's a screenshot from a tutorial I did the other day:
![image](https://user-images.githubusercontent.com/449558/57102039-8755aa80-6cf0-11e9-8607-af3d8848650a.png)

The repr currently includes part of the dataframe, which I find of questionable value to begin with. Additionally, there's ellipsis inside the dataframe, which is hard to see but means that there is actually two different dataframes being shown here (the first part is from the mask of the first transformer, the second from the mask of the second transformer).

I'm not sure what a good solution would be. Possibly a custom repr that hides the columns if they are too long to show? Or trying more magic in the general repr? @NicolasHug had a hard time making that much smarter when he tried in the fall.
Given how central the ColumnTransformer is, I think we should make sure it's easy to understand, including the repr.
user: amueller
issue_13762: state: open
title: doctest SpectralCoclustering fails on Ubuntu arm64
body: ```
=================================== FAILURES ===================================
___________ [doctest] sklearn.cluster.bicluster.SpectralCoclustering ___________
258         The bicluster label of each column.
259 
260     Examples
261     --------
262     >>> from sklearn.cluster import SpectralCoclustering
263     >>> import numpy as np
264     >>> X = np.array([[1, 1], [2, 1], [1, 0],
265     ...               [4, 7], [3, 5], [3, 6]])
266     >>> clustering = SpectralCoclustering(n_clusters=2, random_state=0).fit(X)
267     >>> clustering.row_labels_
Expected:
    array([0, 1, 1, 0, 0, 0], dtype=int32)
Got:
    array([0, 0, 1, 0, 0, 0], dtype=int32)

/usr/lib/python3/dist-packages/sklearn/cluster/bicluster.py:267: DocTestFailure

```

This is 0.20.3 and fails on Ubuntu arm64.
user: xnox
labels: arch:arm: ARM related issues
module:test-suite: everything related to our tests
issue_13761: state: open
title: [MRG] Cross-validation for time series (inserting gaps between the training set and the test set)
body: Time series have temporal dependence, which may cause information leaks during the cross-validation.
One way to mitigate this risk is by introducing **gaps** between the training set and the testing set.
This PR implements such a feature for **leave-p-out**, **K-fold**, and the **naive train-test split**.
As for the walk-forward one, @kykosic is implementing, among others, a similar feature for the class `TimeSeriesSplit` in #13204. I reckon his implementation promising, so I refrain from reinventing the wheel.

Concerning my implementation, I "refactored" the whole structure while still keeping the same public API. `GapCrossValidator` replaces `BaseCrossValidator` and becomes the base where `GapLeavePOut` and `GapKFold` derive from. Although not tested, all other subclasses, I believe, can derive from the new `GapCrossValidator`. I put the quotation marks on the word refactor because I didn't really touch the original code. Instead, my code currently *coexists* with the original one.

### Classes and functions added:
-  GapCrossValidator
    -  GapLeavePOut
    -  GapKFold
-  gap_train_test_split

### Related issues and PRs
#6322, #13204 

### Related users
@kykosic, @amueller, @jnothman, @cbrummitt 

user: WenjieZ
labels: Needs Decision: Requires decision
issue_13756: state: open
title: [idea] extend attributes of RFECV to indicate order of feature elimination
body: Currently sklearn.feature_selection.RFECV provides the scores for the sequence of reduced feature subsets, but it does not expose which feature was eliminated at each step.

this could be useful for semi-automatic procedures and QA/verification.
user: arc12
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_13752: state: open
title: calling ComplementNB.predict_proba with sparse X converts feature_log_prob_ to sparse
body: <!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description

*Surface Level*: `ComplementNB.predict_proba` is slow when you call predict_proba with a sparse X - far slower than if you call predict_proba with a dense X.

*Deeper Problem Description*: `ComplementNB` converts `ComplementNB.feature_log_prob_` to a sparse matrix on every call to predict_proba, which is expensive and wasteful.

Ideally, when BaseNB calls `safe_sparse_dot`, it should do so with matrices that are either both sparse, both dense, or if they are not both the same "kind", then the matrix that gets converted should be cheap to do so.

The unfortunate reality is the current implementation results in the usually large `feature_log_prob_` matrix getting converted to sparse and then thrown away, on every predict_proba call.

Some suggested fixes:

1. Call safe_sparse_dot with the mathematically equivalent `safe_sparse_dot(self.feature_log_prob_, X.T).T` in `_joint_log_likelihood`, which results in X.T getting converted to a dense matrix rather than self.feature_log_prob_ getting converted to a sparse matrix.  This results in a speedup when `X` is much smaller than `feature_log_prob_`

2. Represent `feature_log_prob_` both densely and sparsely, using one or the other for safe_sparse_dot depending on whether X is sparse or dense, or perhaps other considerations  (for example, I noticed that sometimes it is just faster to densify X)

3. Create separate APIs for working with dense vs. sparse X.

Note on 2: There are some technicalities with logsumexp that would need to be resolved.  Calculating logsumexp isn't supported on a sparse matrix because the "log-sum-exp" trick basically makes all entries non-zero, albeit temporarily.  Surely there's a workaround to get logsumexp to work on sparse matrices, I just haven't thought of a solution yet.

Details:
- When the developer calls `Complement.predict_proba`, this calls `BaseNB.predict_log_proba`, which calls `ComplementNB._joint_log_likelihood`, which in turn calls `safe_sparse_dot` on the sparse X and the dense `ComplementNB.feature_log_prob_`
- In `safe_sparse_dot`, if the first argument is sparse and the second is dense, the second argument is converted to a sparse matrix.
- `ComplementNB.feature_log_prob_`  can be expected to be very wide (vocab) and tall (classes) for CBOW-type problems.  
- Thus, the conversion to CSR is very time consuming and costly
- Thus, predict_proba is far slower than it needs to be.

My work-around is to densify X before calling predict_proba.   This works for when len(X) is reasonably small.

However, isn't supporting sparse matrices a major selling point of the sklearn Naive Baye's implementation, and kind of essential for being a good solution?

I've spent some time looking into changing the representation of `MNB.feature_log_prob_` to sparse in my own implementation.  What it comes down to is that a dense/dense multiplication is going to be faster for smaller X but doesn't scale well,, whereas a sparse/sparse multiplication is going to be slower than it needs to be for smaller problems but scales well in both speed and memory.

Moreover, there are technical problems with scipy.special.logsumexp (particularly, getting the "log-sum-exp" trick to work in a sparse context) that I just don't have the time to solve completely.  I do wonder if the log-sum-exp trick is truly necessary in this use case.

Perhaps feature_log_prob_ should be represented both ways and _joint_log_likelihood should choose the multiplication thought to be best give some heuristic?

Or, better yet, perhaps we should fork the naive bayes implementations into ones that support sparse and dense?

#### Steps/Code to Reproduce

Train a ComplementNB model with a large (100's of thousands vocab, ~500+ classes) vocabulary.  Call predict_proba using a sparse X with a single row. Note the execution time.

Call predict_proba with a dense X with a single row.  Note the vastly reduced execution time.

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->

Acceptable performance when X is sparse.




#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->

- With an `X` of 1 x 250K, and a `ComplementNB.feature_log_prob_` of 503 x 250K , the combined cost of sparsifying `feature_log_prob_` and performing `safe_sparse_dot` on the result is in the neighborhood of 4 seconds. With dense X, 0.088 seconds.
- The sparsification of `ComplementNB.feature_log_prob_` is repeated on every call to `predict_proba`, rather than once or never.




#### Versions


<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->

System:
    python: 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)]
executable: C:\git\brightfield-jobtitle-classification\.venv\Scripts\python.exe
   machine: Windows-10-10.0.16299-SP0

BLAS:
    macros:
  lib_dirs:
cblas_libs: cblas

Python deps:
       pip: 19.1
setuptools: 41.0.1
   sklearn: 0.20.1
     numpy: 1.15.4
     scipy: 1.2.1
    Cython: None
    pandas: 0.23.4

<!-- Thanks for contributing! -->

user: kyle-pena-nlp
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_13733: state: open
title: Ambiguous way to store n-grams in TfidfVectorizer and CountVectorizer
body: #### Description

I'm trying to convert TfidfVectorizer into ONNX and it is not always to possible to find the exact list of tokens which composes a n-grams. I'd like to store n-grams as tuple instead of strings to clear ambiguities in ``vocabulary_``. The notebook explains it all: 
https://github.com/sdpython/mlinsights/blob/master/_doc/notebooks/sklearn/traceable_ngrams_tfidf.ipynb

#### Steps/Code to Reproduce
https://github.com/sdpython/mlinsights/blob/master/_doc/notebooks/sklearn/traceable_ngrams_tfidf.ipynb

#### Expected Results
No change except in interval members vocabulary_.

#### Versions
< 0.21

user: sdpython
labels: Needs Decision: Requires decision
issue_13727: state: open
title: Is there any possibility to visualize the SGDClassifier results?
body: I was trying to implement a classification problem by SGDClassifier and used the following codes:

SGDClassifier(penalty='elasticnet',random_state=seed,shuffle=False,loss='log')

i.e., I use a Logistic Regression for this classification problem and deploy both l1 and l2 regularized terms (elasticnet) to this model. Now the problem is I want to visualize the result like the enet_path in the sklearn (please see the following picture)
![image](https://user-images.githubusercontent.com/39890732/56554838-9c7e5b00-65c5-11e9-9d7d-4c8c7921e675.png)

As far as I know, the enet_path is for ElasticNet regression, not the classification problem that I want. So is there any method to realize this problem?

By the way, I have tried glmnet package which can conveniently solve this problem in R. Unfortunately, this package could not install in Python under my Win10 environment (need Linux).



<Version: For scikit-learn >= 0.20:>


user: Minqi824
issue_13714: state: open
title: 2D sample weight for MultiOutputEstimator
body: #### Reference Issues/PRs
Fixes #10912.

#### What does this implement/fix? Explain your changes.
The original issue #10912 is about 2D sample weight for MultiOutputRegressor.
I made modifications in MultiOutputEstimator, so that this feature is available for not only MultiOutputRegressor but also MultiOutputClassifier.

#### Any other comments?
2D sample weight is useful in my use case, because different outputs have different numerical ranges.

user: kevin1kevin1k
issue_13674: state: open
title: Strange behavior of label_binarize when there's only one class in y
body: When there's only one class in y, label_binarize produces unexpected result
```
from sklearn.preprocessing import label_binarize
label_binarize([1, 1, 1], classes=[1])
# actual result [[0], [0], [0]]
# expected result [[1], [1], [1]]
label_binarize([1, 1, 1], classes=[0])
# actual result [[0], [0], [0]]
# expected result [[0], [0], [0]]
```
version: 0.20.3 and 0.21 dev
user: qinhanmin2014
issue_13669: state: open
title: Fuzzy c-means clustering algorithm is partly implemented.
body: I want to resolve the [WIP] Fuzzy c-means clustering #9327 in a nice way with few commits.

I implement the Fuzzy c-means clustering in a concise way. And in order to make the algorithm continuable, I copy the frame of k_means_.py and implement FCM algorihtm based on it.

One more thing, it is the first time that I try to be a contributor of a public repository, so probably I have made a lot of mistakes. Hope you(someone may concern) can help me to point them out.

Thanks again and Enjoy the Code.

Good days.

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: Sean16SYSU
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_13668: state: open
title: Support for Pandas DataFrames in kernels & heterogeneous kernel
body: #### Description

Categorical features in datasets are commonly treated with One-Hot-Encoding. However, a Gram matrix of a dot kernel can be calculated much faster directly with:

```python
def nominal_gram_dot(X1, X2):
   gram_matrix = np.zeros((X1.shape[0], X2.shape[0]))
   for i in range(X1.shape[0]):
       gram_matrix[i, :] = np.sum(X1[i, :] == X2, axis=1)
   return gram_matrix
```

On heterogeneous data, which contain both, numerical and categorical features, the Gram matrix can be then computed with:

```python
def hetero_kernel(X1, X2):
   X1_cat, X1_num = split_data(X1, categoricals)
   X2_cat, X2_num = split_data(X2, categoricals)
   gram_cat = nominal_gram_dot(X1_cat, X2_cat)
   gram_num = np.dot(X1_num, X2_num.T)
   gram = gram_cat + gram_num
   return gram
```

This can be used as a combination not only for the linear kernel but also polynomial, RBF (while defining "categorical Euclidean distance"), etc..

#### Comparison

![runtime-comp-samp](https://user-images.githubusercontent.com/30484650/56349073-1ed2dd80-61c8-11e9-9d84-e2af0225bba6.png) In this graph I compare np.dot after one-hot-encoding versus my implementation. Done on synthetic data.

#### Steps/Code to Reproduce

```python
n_features = 50
n_samples = range(300, 2400, 100)
cardinality = 10
np.random.seed(seed=21)

times_samp = list()
for samples in n_samples:
   df = pd.DataFrame(np.random.randint(cardinality, size=(samples, n_features)))
   for col in df.columns:
       df[col] = df[col].astype('category')

   start = time.time()
   df_ohe = pd.get_dummies(df, dummy_na=True, sparse=False)
   gram_ohe = np.dot(df_ohe, df_ohe.T)
   t_ohe = time.time() - start

   start = time.time()
   gram_delta = ker.build_hetero_kernel(np.ones(n_features, dtype='bool'))(X, X)
   t_delta = time.time() - start

   np.testing.assert_array_equal(gram_delta, gram_ohe)
   times_samp.append([t_delta, t_ohe])

times_samp = np.array(times_samp)
```

This is how you can reproduce the results of the linked graph.

#### Versions

System:
   python: 3.7.2 (default, Feb 12 2019, 08:15:36)  [Clang 10.0.0 (clang-1000.11.45.5)]
   machine: Darwin-18.5.0-x86_64-i386-64bit

BLAS:
   cblas_libs: openblas, openblas

Python deps:
   sklearn: 0.20.3
   numpy: 1.16.2
   scipy: 1.2.1
   Cython: None
   pandas: 0.24.2

##### Do you think I can make a PR for this issue? 
Maybe a contribution to the `sklearn.metrics.pairwise` module?
user: 66Antigravity
issue_13666: state: open
title: Unclear behavior of max_train_size argument in TimeSeriesSplit
body: #### Description

So I am trying to understand the behavior of [`TimeSeriesSplit`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html#sklearn.model_selection.TimeSeriesSplit). Especially the `max_train_size` parameter. I was initially surprised that it is an absolute number and not a ratio like it is in other splitting operations.

I traced this parameter to issue #8249 and PR #8282 and I realized that it was added to support window-based splitting, as it is described [here](https://topepo.github.io/caret/data-splitting.html#data-splitting-for-time-series). This was very surprising for me because this is not really clear from documentation that this is happening. Moreover, I found parameters `initialWindow`, `horizon`, and `fixedWindow` much easier to understand, especially with that image.

I would suggest that:
* Documentation is improved here. Such visualization as shown in https://topepo.github.io/caret/data-splitting.html#data-splitting-for-time-series would really help a lot.
* We consider using or sample based parameters, like `initialWindow`, `horizon`, and `fixedWindow`, or ratio/fold based ones, but not both, because it is very confusing.

If we have splitting done by number of folds (which I prefer because it makes things adapt to different dataset sizes automatically), then also window size should be expressed in folds. In a way, parameters could then be:
* How many folds to do.
* Number of folds used in horizon, i.e., used in test data. It looks like this is currently fixed to 1 in this splitting operation and cannot really be configured. I suggest we allow this to be configured.
* Number of folds used in the window, i.e., training data. Default could be None, which would mean a non-fixed window and would mean to use all folds before the test data. Or you could fix it to get a sliding window.

#### Versions

Relates to how it is in sklearn v0.20.3.
user: mitar
labels: Needs Decision: Requires decision
issue_13658: state: open
title: Modify check_estimator to handle fit() additional keywords?
body: #### Description
I'm creating a library that will hopefully be sklearn compatible eventually. I'd like to test the compatibility of my estimators using the ```sklearn.utils.estimator_checks.check_estimator()``` function. However, when passing in an estimator whose fit() method has additional keywords, ```check_estimator()``` breaks. From [here](https://scikit-learn.org/stable/developers/contributing.html#fitting) it seems that additional keywords to ```fit()``` are acceptable to have with the sklearn api so it would be nice if ```check_estimator()``` supported that as well. I apologize if this is the wrong place to put it, I can move it if necessary.

#### Steps/Code to Reproduce
Use ```check_estimator()``` on a custom class that inherits from ```sklearn.base.BaseEstimator```. I'm sorry I can't post code!

#### Expected Results
No error is thrown if all the checks pass, error thrown if checks don't pass

#### Actual Results
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-4444863e9af0> in <module>
----> 1 check_estimator(mod)

/anaconda3/lib/python3.7/site-packages/sklearn/utils/estimator_checks.py in check_estimator(Estimator)
    302     for check in _yield_all_checks(name, estimator):
    303         try:
--> 304             check(name, estimator)
    305         except SkipTest as exception:
    306             # the only SkipTest thrown currently results from not

/anaconda3/lib/python3.7/site-packages/sklearn/utils/testing.py in wrapper(*args, **kwargs)
    348             with warnings.catch_warnings():
    349                 warnings.simplefilter("ignore", self.category)
--> 350                 return fn(*args, **kwargs)
    351 
    352         return wrapper

/anaconda3/lib/python3.7/site-packages/sklearn/utils/estimator_checks.py in check_estimators_dtypes(name, estimator_orig)
   1093         estimator = clone(estimator_orig)
   1094         set_random_state(estimator, 1)
-> 1095         estimator.fit(X_train, y)
   1096 
   1097         for method in methods:

TypeError: fit() missing 1 required positional argument: 'W'
```

#### Versions
```
System:
    python: 3.7.1 (default, Dec 14 2018, 13:28:58)  [Clang 4.0.1 (tags/RELEASE_401/final)]
executable: /anaconda3/bin/python
   machine: Darwin-18.5.0-x86_64-i386-64bit

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: /anaconda3/lib
cblas_libs: mkl_rt, pthread

Python deps:
       pip: 18.1
setuptools: 40.6.3
   sklearn: 0.20.1
     numpy: 1.15.4
     scipy: 1.1.0
    Cython: 0.29.2
    pandas: 0.23.4
```

user: madvoid
issue_13656: state: open
title: MLPRegressor, warm_start not working as expected
body: I think I found a bug when it comes to the MLPRegressor (in v.0.20.3), especially with "warm_start" set to true. The problem occured to me, as well as to other people I know. 

We have a dataset with some training data, and try to train the NN, with MLPRegressor, of 5000 iterations, 50 hidden neurons in one layer and the fixed random seed 0. 
```

nn = MLPRegressor(activation='logistic', solver='lbfgs', max_iter=5000, hidden_layer_sizes=(50,), alpha=0, random_state=0)
nn.fit(x_train, y_train)
```

This works as expected, however if we create an MLPRegressor with 1 iteration, but we are calling "fit" in a loop over a range of 5000, the function we have learned looks totally wrong, since it is totally inaccurate with respect to the trainingsset.

```
nn = MLPRegressor(activation='logistic', solver='lbfgs', max_iter=1,hidden_layer_sizes=(50,), alpha=0, random_state=0, warm_start=True)
for i in range(0,5000):
  nn.fit(x_train,y_train) 
```

We tried calculating the mean squared error durring each iteration of the second code snippet, and we calculated the MSE after the first code snippet. None of the MSEs, calculated during the iterations, was even close to the MSE we calculated after the first code snippet. 

Since on the scikit-learn website it is stated, that warm start can be used for monitoring, I am pretty sure that this is a bug, since the end results are definitely not the same. 
user: leReppiz
issue_13654: state: open
title: Common test for memmap when fitting estimators should use appriopriate dtype
body: The common tests `check_classifier_train` and `check_regressor_train` with `readonly_memmap=True` use float64 data. However, all tree based estimators requires float32 data, leading to a copy and resulting in not actually testing on read-only memmap.

We could either run the tests with both dtypes, but it would make the test twice as long, or add a new estimator tag holding with dtype is copying.
user: jeremiedbb
labels: Needs Decision: Requires decision
issue_13631: state: open
title: Do not require lexicographic order of classes_ and assume max()=positive class
body: I'm implementing a classifier[1] which does *concept learning*, i.e. unsymmetrical binary classification with a (sometimes user-defined) *target/positive class* and the learned model describing that class, while everything not matching the model is classified as the other "negative" class.

I'm stumbling across the documented[2] assumption that `classes_` be sorted lexicographically and the positive class has to be the last entry, which is clearly at odds with my background.

https://github.com/scikit-learn/scikit-learn/issues/4450#issuecomment-86298663 gave me the idea to wrap the classifier and translate class values, which works, but I consider it a workaround: Therefore the request to drop that assumption.


maybe related: #8100
sklearn v0.20.3

---

[1] <https://github.com/azrdev/sklearn-seco/>
[2] Here it went into the glossary: <https://github.com/scikit-learn/scikit-learn/pull/9517/files#r152116415>
user: azrdev
labels: Needs Decision: Requires decision
issue_13625: state: open
title: FIX check metric parameters of the given object in neighbors
body: Fixes #4194

Checks the metric parameters of the given tree object and raise an error if the metric params of the given object and the self don't match.
user: adrinjalali
issue_13619: state: open
title: Shuffled GroupKFold
body: I agree that it's right for GroupKFold to try to balance the folds, since it's mandatory for certain scenarios. However, since this is done deterministically, it makes repeated GroupKFold impossible, since even if you shuffle the groups, the folds you get might be the same, for example, sizes of 5, 4, 3 in 2 folds will always be assigned (5, ) and (4, 3).

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/model_selection/_split.py#L506

My idea would be, rather than selecting the largest each time and assigning it to the lightest fold, select a random group, weighted by size, so large groups are more likely to be assigned first and less likely to be in the same fold as each other.

It might also be interesting to have an unbalanced option for GroupKFold for cases when you know that your groups are of similar sizes.

What do you think?

There was a previous feature request for this: https://github.com/scikit-learn/scikit-learn/issues/13148. It's also kind of similar/related to https://github.com/scikit-learn/scikit-learn/issues/9297.
user: jamesmyatt
issue_13614: state: open
title: instantiating a ColumnTransformer with already fitted transformers
body: Can I instantiate a ColumnTransformer with already fitted transformers? I get an error right now saying the `fit` wasn't called on `ColumnTransformer`. I don't want to call `fit` on the transformer since I am using already fitted transformers and just want to use `ColumnTransformer` to aggregate the columns into a matrix.
user: sebastiangonsal
issue_13608: state: open
title: using compute_sample_weight for the fit_params of cross_val_predict
body: I would like to know if it is possible to use "compute_sample_weight" within "cross_val_predict" to provide sample weights in fit_params?

I'm using the following code:
`y_pred = cross_val_predict(clf, X, y, cv = cv, method = 'predict', fit_params = {'sample_weight':compute_sample_weight("balanced", y)})`

But I think that it is not correct because `compute_sample_weight("balanced", y)` computes weights for all labels, whereas I would like that this function successively applied on each training set generated by the split operated by cross_val_predict. I think that the parameter "indices" of compute_sample_weight() could be a solution, but I don't know how to apply it.
user: AlexandraBomane
issue_13587: state: open
title: BaggingClassifier uses Class Label as Index to Array when Voting
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
BaggingClassifier uses Class Label as Index to Array when Voting

#### Steps/Code to Reproduce

Provide a base estimator to _BaggingClassifier_ that does not define the function _predict_proba_. This results in _BaggingClassifier_ resorting to voting. It appears the code for performing voting uses class labels as array indices instead of looking up the index of the class label in the _classes__ member.

Example:
```python
import numpy as np
from sklearn.ensemble import BaggingClassifier

class Foo:
    
    def __init__(self):
        pass
    
    def fit(self, X, Y, W=None):
        return self
    
    def predict(self, X):
        return np.full(X.shape[0], True, np.bool)
    
    def score(self, X, Y):
        YH = self.predict(X)
        return (Y == YH).mean()
    
    def get_params(self, deep=True):
        return {}
    
    def set_params(self, **params):
        for k, v in params:
            setattr(self, k, v)
        return self
    
# %%
A = np.random.rand(10, 4)
Y = np.random.randint(2, size=10, dtype=np.bool)
bc = BaggingClassifier(Foo())
bc.fit(A, Y)
YH = bc.predict(A)
print('BaggingClassifier Voting Result: ')
print(YH)
print('Ensemble Member Predictions: ')
for Ei in bc.estimators_:
    print(Ei.predict(A))
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
In the above code snippet, _BaggingClassifier_ should return an array of _True_ since it is the majority prediction of all ensemble members.

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
_BaggingClassifier_ returns an array of False. This issue only occurs when the base estimator does not define the function _predict_proba_.

The issue appears to be due to lines 137 and 140 in ensemble/bagging.py.

https://github.com/scikit-learn/scikit-learn/blob/e14ac6d36d9dd069cc7fc2e51e4973514a003591/sklearn/ensemble/bagging.py#L137

https://github.com/scikit-learn/scikit-learn/blob/e14ac6d36d9dd069cc7fc2e51e4973514a003591/sklearn/ensemble/bagging.py#L140

The predictions of the ensemble members are directly used as indices into the original array. I'm guessing the prediction labels need to be converted into class labels using _estimator.classes__.

#### Versions
System:
    python: 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)]
executable: C:\Users\XXXXXX\Anaconda3\pythonw.exe
   machine: Windows 2012 ServerR2

BLAS:
    macros: 
  lib_dirs: 
cblas_libs: cblas

Python deps:
       pip: 18.1
setuptools: 40.6.3
   sklearn: 0.20.1
     numpy: 1.15.4
     scipy: 1.1.0
    Cython: 0.29.2
    pandas: 0.23.4

<!-- Thanks for contributing! -->

user: nogilnick
issue_13570: state: open
title: The default value of some parameters make little sense
body: See #13541 
Seems that we've decided that all parameters should have a default. (Can someone tell me why? I don't like it.) For some parameters, the default value makes little sense (e.g., n_clusters in KMeans, eps in DBSCAN), I think we should have a way to recommend users to set these parameters manually.
My proposal is to define a DefaultMeaninglessWarning and raise it when these parameters are not set manually.
user: qinhanmin2014
issue_13557: state: open
title: cross_validate hang randomly when training svc with polynomial kernel.
body: #### Description
`cross_validate` hang randomly when training svc with polynomial kernel.

#### Steps/Code to Reproduce
```python
from sklearn import datasets
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split, cross_validate

iris = datasets.load_iris()
x = iris.data
y = iris.target
X_choose, x_test, y_choose, y_test = train_test_split(x, y, test_size=0.3)

params = [[0.6652997139930452, 'poly', 7, 4178.386000737241],
          [1.2346990434544882, 'poly', 7, 4317.581190465473],
          [0.8156943235551155, 'poly', 8, 864.1583649816441]]

for c, kernel, degree, gamma in params:
    clf = SVC(C=c, kernel=kernel, degree=degree, gamma=gamma)
    cv_results = cross_validate(clf, X_choose, y_choose, cv=5,
                                return_train_score=False)
    print(cv_results['test_score'].mean())
```

#### Expected Results
Three scores should be shown.

#### Actual Results
Sometimes hang.
I waited for few minutes.
Also I checked running code line doesn't change using gdb.
Backtrace is [here](https://gist.github.com/sam-yusuke/7099829b1797d6867eb928264597979d)
#### Versions
```
>>> import sklearn; sklearn.show_versions()
System:
    python: 3.7.3 (default, Mar 27 2019, 22:11:17)  [GCC 7.3.0]
executable: /home/yusuke/miniconda3/envs/py37_automl_examples2/bin/python
   machine: Linux-4.20.0-042000-generic-x86_64-with-debian-buster-sid

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: /home/yusuke/miniconda3/envs/py37_automl_examples2/lib
cblas_libs: mkl_rt, pthread

Python deps:
       pip: 19.0.3
setuptools: 40.8.0
   sklearn: 0.20.3
     numpy: 1.16.2
     scipy: 1.2.1
    Cython: None
    pandas: None
```

Thank you!
user: xiong-jie-y
issue_13556: state: open
title: More details about the Crammer-Singer method for LinearSVM
body: #### Description

On the current [page about the LinearSVC model](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC), we are given the option to use either OVR or Crammer-Singer. Though the former is a well-known method, it seems that Crammer-Singer is a very specific algorithm developed for SVMs. The page talks about the nicer theoretical foundations of the method, as well as the longer training time. However, the information about the method is a bit sparse (more about it [here](https://scikit-learn.org/stable/modules/svm.html), but it says basically the same thing). It would be nice to have:

* A link to the [original paper](http://jmlr.csail.mit.edu/papers/volume2/crammer01a/crammer01a.pdf)
* A general overview of the differences between Crammer-Singer and OVR/OVO
* A benchmark comparison between the methods above, in terms of computation time and accuracy, on varying dataset sizes
* An explanation of the important formulas from the paper

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
scikit-learn 0.20.3

<!-- Thanks for contributing! -->

user: xhluca
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_13555: state: open
title: Refactor BaseEstimator.get_params for easier use in subclasses
body: Below is a proposal for a minor refactoring of `BaseEstimator.get_params` to make it easier to use in subclasses. Would like to hear your opinions :)
If it sounds like a sensible idea, I would be glad to throw a PR.

### Issue
Currently, invoking `get_params` from custom subclasses can produce an incomplete list of parameters, depending on how you define the subclass `__init__`.

For example, if you have, say, the following class:

```python
from sklearn.base import BaseEstimator


class SomeSKLearnEstimator(BaseEstimator):
    def __init__(self, param1=1, param2='foo'):
        self.param1 = param1
        self.param2 = param2
```

and want to make some subclass from it. You might want to do something like:

```python
class MySubclass(SomeSKLearnEstimator):
    def __init__(self, myparam=42, **kwargs):
        super().__init__(**kwargs)
        self.myparam = myparam
```

This however produces an incomplete parameter dict when calling `get_params()`, which can be surprising:
        
```python     
x = MySubclass(myparam=42, param1=2, param2='bar')
print(x.get_params())
# {'myparam': 42}
```

You could work around it by copy-and-pasting the keyword arguments from the parent class and passing them via `super()`:

```python
class MySubclass(SomeSKLearnEstimator):
    def __init__(self, myparam=42, param1=1, param2='foo'):
        # These params can be a lot!
        # Duplicated code.
        super().__init__(param1=param1, param2=param2)
        self.myparam = myparam


x = MySubclass(myparam=42, param1=2, param2='bar')
print(x.get_params())
# {'param1': 2, 'param2': 'bar', 'myparam': 42}
```

However, the parameters can be many, and this leads to code duplication.

### Proposal
Refactor `BaseEstimator.get_params` so that client code has access to the parent class' `__init__` and has more control over how to use it together with its own subclass parameters. Specifically:

1. Make `_get_param_names` a static method that accepts a class.
2. Extract the core logic of `get_params` into an auxiliary `_get_params_from`

This would look like this:

```python
class BaseEstimator:
    """Base class for all estimators in scikit-learn
    Notes
    -----
    All estimators should specify all the parameters that can be set
    at the class level in their ``__init__`` as explicit keyword
    arguments (no ``*args`` or ``**kwargs``).
    """

    @staticmethod
    def _get_param_names(klass):
        """Get parameter names for the estimator"""
        # fetch the constructor or the original constructor before
        # deprecation wrapping if any
        init = getattr(klass.__init__, 'deprecated_original', klass.__init__)
        if init is object.__init__:
            # No explicit constructor to introspect
            return []

        # introspect the constructor arguments to find the model parameters
        # to represent
        init_signature = inspect.signature(init)
        # Consider the constructor parameters excluding 'self'
        parameters = [p for p in init_signature.parameters.values()
                      if p.name != 'self' and p.kind != p.VAR_KEYWORD]
        for p in parameters:
            if p.kind == p.VAR_POSITIONAL:
                raise RuntimeError("scikit-learn estimators should always "
                                   "specify their parameters in the signature"
                                   " of their __init__ (no varargs)."
                                   " %s with constructor %s doesn't "
                                   " follow this convention."
                                   % (klass, init_signature))
        # Extract and sort argument names excluding 'self'
        return sorted([p.name for p in parameters])
    
    def _get_params_from(self, names, deep=True):
        """Get parameters for this estimator from the given names"""
        out = dict()
        for key in names:
            value = getattr(self, key, None)
            if deep and hasattr(value, 'get_params'):
                deep_items = value.get_params().items()
                out.update((key + '__' + k, val) for k, val in deep_items)
            out[key] = value
        return out

    def get_params(self, deep=True):
        """Get parameters for this estimator.
        Parameters
        ----------
        deep : boolean, optional
            If True, will return the parameters for this estimator and
            contained subobjects that are estimators.
        Returns
        -------
        params : mapping of string to any
            Parameter names mapped to their values.
        """
        # The source code now also makes the get_params recipe obvious
        names = self._get_param_names(self.__class__)
        return self._get_params_from(names, deep=deep)
    
    # ... rest of the class is abbreviated ...
```

This way, client code could do something like this:

```python
class MySubclass(SomeSKLearnEstimator):
    def __init__(self, myparam=42, **kwargs):
        # Just pass **kwargs.
        # No duplicated code.
        super().__init__(**kwargs)
        self.myparam = myparam
        
    def get_params(self, deep=True):
        # Gotta override get_params. But it shouldn't be hard.
        
        # params from parent
        parent_names = self._get_param_names(super())
        params = self._get_params_from(parent_names, deep=deep)
        
        # Add my params
        names = self._get_param_names(self.__class__)
        params.update(self._get_params_from(names, deep=deep))
        # or just simply
        # params['myparam'] = self.myparam
        
        return params


x = MySubclass(myparam=42, param1=2, param2='bar')
print(x.get_params())
# {'param1': 2, 'param2': 'bar', 'myparam': 42}
```

### Motivation
- Personal motivation:
    - In the current project I'm working on, I would like to extend some sklearn classes via cooperative multiple inheritance where I want to override the `__init__` method, but still want to `get_params` get the params from the parent class.
- Also, other people seem to have encountered the same problem:
    1. [here](https://github.com/scikit-learn/scikit-learn/issues/8822)
    2. [and here](https://stackoverflow.com/questions/51430484/how-to-subclass-a-vectorizer-in-scikit-learn-without-repeating-all-parameters-in)
 
user: alegonz
labels: Needs Decision: Requires decision
issue_13501: state: open
title: Disregard NaNs in Normalizer
body: Related to #10404
I think we should support NaNs in ``Normalizer``/``normalize``
user: qinhanmin2014
labels: Needs Decision: Requires decision
issue_13492: state: open
title: Added vip_ attribute to PLSRegression
body: #### Reference Issues/PRs
#7050 (PLSRegression VIP score calculation)

#### What does this implement/fix? Explain your changes.
Adds the Variable Importance in Projection (VIP) score to PLSRegression (as vip_ attribute)


user: KevinMMendez
issue_13490: state: open
title: Tree export fill colors
body: #### What does this implement/fix? Explain your changes.

This fixes the conversion of rgb to hexstrings in `sklearn.tree.export` and
adds the possibility for the user to supply his/her own colors.


```
from sklearn.tree.export import plot_tree
from sklearn.datasets import make_moons
from sklearn.tree import DecisionTreeClassifier
import matplotlib.pyplot as plt

X, y = make_moons()
tree = DecisionTreeClassifier()
tree.fit(X, y)

plt.figure(dpi=300)
plot_tree(tree, fill_colors=['xkcd:red', 'xkcd:sky'], filled=True)
plt.savefig('tree.png')

```

![tree](https://user-images.githubusercontent.com/5488440/54769220-11dec080-4c01-11e9-9d6c-4ce172852460.png)

user: maxnoe
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_13482: state: open
title: Regression metrics - which strategy ?
body: I recently came across #12895 (with PR #13467) and the older #6457, this woke up an old topic that I would like to share.

In our team, we had the need to provide model performance metrics, for regression models. This is a slightly different goal than using metrics for grid-search or model selection. Indeed the metric is not only used to "select the best model" but to provide users with feedback about "how good a model is".

For regression models I introduced three categories of metrics that happened to be quite intuitive:

 * Absolute performance (L2 RMSE, L1 MAE): these metrics can all be interpreted as an "average prediction error" ("average" in the broad sense here) expressed in the unit of the prediction target (e.g. "average error of 12kWh")

 * Relative performance (L2 CVRMSE, L1 CVMAE, and per-point relative metrics such as MAPE or MARE, MARES, MAREL...): these metrics can all be interpreted as an "average relative prediction error" expressed as a percentage of the target (e.g. "average error of 10%").

 * Comparison to a dummy model (L2 RRSE, L1 RAE): these metrics can all be interpreted as a ratio between the performance of the model at hand, and the performance of a dummy, constant model (predicting always the average). These need to be inverted to be intuitive e.g. "20% -> 5 times better than a dummy model"

Of course these categories are "applicative". They all make sense from a user point of view, however as far as model selection is concerned, only two make sense (MAE and RMSE). Not even R² because R²=1-RRSE² so it is not a performance metric but a comparison to dummy metric (but I dont want to open the debate here so please refrain from objecting on that one :) ).

Anyway my question for the core `sklearn` team is: shall I propose a pull request with all these metrics ? I'm ready to shoot since we've done it in our private repo, aligned with sklearn `regression.py` file. So it is rather a matter of deciding if this is a good idea. And if so, introducing categories might be needed, to help users better understand.

An alternative might be to create a small independent projet containing all the metrics, leaving only the `mean_absolute_error` (L1) and `mean_squared_error` (L2) in sklearn.

Any thoughts on this ?
user: smarie
issue_13478: state: open
title: Add `return_predictions` option to the model_selection.cross_validate() API
body: The `cross_validate()`, `cross_val_predict()`, and `cross_val_score()` methods from the `model_selection` module provide highly compact and convenient API for most daily works. Yet often times, one would like to get 1) multiple scores for both train and test, and 2) predictions for all samples using cross validation, which is an aggregation of test predictions from all folds.

Current API doesn't seem to allow both goals to be achieved in a single cv run. Is it possible to add something like a `return_predictions` option to the model_selection.cross_validate() API, as in [here](https://stackoverflow.com/a/53391623), to simply allow access to the results (which I believe have already been computed inside the function)?

Thank you!
user: herrlich10
issue_13438: state: open
title: Estimator Tags todo / proposed tags
body: Finally Estimator tags are merged (#6599, #8022) but there's still a lot to do.
We need to decide which tags to add and also use the tags more in the tests.
The following tags were proposed and I think might be useful:

- supports sparse data
- binary-only classifier
- multi-label only classifier
- positive data only
- supports inverse transform?
- support sample weights?
- 1d inputs only?

The ``X_types`` field is not really thought through and we need to come up with a more general mechanism to generate input data for an estimator for testing.

We might also want to remove the ``_pairwise`` property and replace it by a tag.
user: amueller
issue_13411: state: open
title: Add new manifold learning algorithm UMAP?
body: #### Description
Is there any plan to include new algorithms like this https://github.com/lmcinnes/umap?

I know they already have the implementation. Maybe it's good to include this in sklearn as well?

McInnes, L, Healy, J, UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, ArXiv e-prints 1802.03426, 2018

user: Kulbear
issue_13408: state: open
title: Add Dynamic Mode Decomposition to sklearn.decomposition?
body: Hi everyone,

Since it was first proposed in 2008 by P. Schmid ([Dynamic mode decomposition of numerical and experimental data, _J. Fluid Mech._, 2008](https://hal-polytechnique.archives-ouvertes.fr/hal-01020654/file/DMS0022112010001217a.pdf)) in the field of fluid dynamics, Dynamic Mode Decomposition (DMD) has become increasingly popular for dimensionality reduction of dynamical systems and time-series modeling. Since then, numerous algorithms have been proposed to compute DMD and it is only recently that Héas & Herzet ([Optimal low-rank DMD, arXiv, 2017](https://arxiv.org/abs/1701.01064)) have shown that a closed-form optimal solution exists and that its computational complexity is similar to that of PCA.

Various Python packages implement DMD (e.g. [pyDMD](https://github.com/mathLab/PyDMD) or [modred](https://github.com/belson17/modred)) but, to the best of my knowledge, none of these uses the sklearn api. Because of its increasing popularity and because, deep down, the algorithm ain't much more complicated than PCA, I was wondering if there are currently any plan to add DMD to the set of linear dimensionality reduction techniques offered by scikit-learn?

Thanks a lot :)
user: loiseaujc
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_13407: state: open
title: No parallelization of predict_proba method of MultiOutputClassifier
body: #### Description

The `predict` method of `sklearn.multioutput.MultiOutputEstimator` is (optionally) parallelized. The `predict_proba` method of `sklearn.multioutput.MultiOutputClassifier` does not allow an optional parallelization, using a list comprehension to generate predictions.

[predict](https://github.com/scikit-learn/scikit-learn/blob/ab649266e115ef61d704fee3eae1f1b2fa056141/sklearn/multioutput.py#L193)

[predict_proba](https://github.com/scikit-learn/scikit-learn/blob/ab649266e115ef61d704fee3eae1f1b2fa056141/sklearn/multioutput.py#L320)

##### Versions
master branch

user: georgedouzas
issue_13405: state: open
title: correlation feature selection?
body: I wonder whether we should add unsupervised feature selection using correlation. It's a strategy that's very common in statistics and might just have it for completeness.
It's certainly more useful than the variance based one, which I think is useless for any threshold != 0.
user: amueller
issue_13403: state: open
title: `pairwise_distances` produces wrong output for asymmetric similarity measures
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
The `pairwise_distances` function implicitly assumes that any distance function (or provided callable) is _symmetric_ (i.e. a statistical _metric_). Hence, if given an asymmetric measure, such as KL-Divergence or any of the inclusion measures commonly used in NLP (see e.g. Section 3.2 [here](http://aclweb.org/anthology/E17-1007)), only the values **above** the diagonal in the returned distance matrix are correct.

Ideally, there would be a flag in `pairwise_distances` that forces the explicit calculation of the whole distance matrix. If this is considered to be too much of an edge case, then it might be good to update the documentation to more accurately reflect the behaviour of `pairwise_distances` (I am aware the package is called `metrics`, though this is perhaps too subtle a hint). 

#### Steps/Code to Reproduce
```
import numpy as np
from scipy.stats import entropy
from sklearn.metrics import pairwise_distances

X = np.array([[2, 1, 1, 5], [2, 1, 1, 1], [1, 3, 5, 1], [1, 2, 2, 1], [1, 1, 2, 3]])

D = pairwise_distances(X, metric=entropy)
```
#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
Explicit pairwise calculation results in
```
D_kl = np.zeros((5, 5))
for idx, x in enumerate(X):
    for jdx, xx in enumerate(X):
        D_kl[idx, jdx] = entropy(x, xx)

print(D_kl)

array([[0.        , 0.30634551, 0.85263093, 0.48866707, 0.10949394],
       [0.26589908, 0.        , 0.42879601, 0.18232156, 0.25537921],
       [0.79868362, 0.37184074, 0.        , 0.06895927, 0.32119288],
       [0.48379905, 0.16425203, 0.07024034, 0.        , 0.20209769],
       [0.13141019, 0.23338263, 0.40876682, 0.21766213, 0.        ]])
```
#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
```
print(D)

array([[0.        , 0.30634551, 0.85263093, 0.48866707, 0.10949394],
       [0.30634551, 0.        , 0.42879601, 0.18232156, 0.25537921],
       [0.85263093, 0.42879601, 0.        , 0.06895927, 0.32119288],
       [0.48866707, 0.18232156, 0.06895927, 0.        , 0.20209769],
       [0.10949394, 0.25537921, 0.32119288, 0.20209769, 0.        ]])

# Perhaps more explicit, D - D_kl should return an all zero matrix, but instead returns
print(D - D_kl)

array([[ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ],
       [ 0.04044643,  0.        ,  0.        ,  0.        ,  0.        ],
       [ 0.05394731,  0.05695527,  0.        ,  0.        ,  0.        ],
       [ 0.00486802,  0.01806952, -0.00128107,  0.        ,  0.        ],
       [-0.02191625,  0.02199659, -0.08757394, -0.01556444,  0.        ]])
```
#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
System:
    python: 3.6.4 (default, Mar  1 2018, 18:36:42)  [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
executable: /Users/thomas/.virtualenvs/nlp/bin/python
   machine: Darwin-16.7.0-x86_64-i386-64bit

BLAS:
    macros: NO_ATLAS_INFO=3, HAVE_CBLAS=None
  lib_dirs: 
cblas_libs: cblas

Python deps:
       pip: 19.0.1
setuptools: 28.8.0
   sklearn: 0.20.2
     numpy: 1.16.0
     scipy: 1.2.0
    Cython: 0.28.3
    pandas: 0.23.1

<!-- Thanks for contributing! -->

user: tttthomasssss
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_13399: state: open
title: Unintuitive results in make_regression
body: Adding more samples makes the data less well-specified, which I think is really weird:

```python
from sklearn.metrics import r2_score
from sklearn.datasets import make_regression
X, y, coef = make_regression(n_samples=10000, n_features=10, n_informative=4, effective_rank=1, noise=1, coef=True, shuffle=False, tail_strength=0.01, random_state=42)
r2_score(y, np.dot(X, coef))
```
> 0.003

```python
X, y, coef = make_regression(n_samples=100, n_features=10, n_informative=4, effective_rank=1, noise=1, coef=True, shuffle=False, tail_strength=0.01, random_state=42)
r2_score(y, np.dot(X, coef))
```
> 0.95
user: amueller
labels: Needs Decision - Close: Requires decision for closing
issue_13396: state: open
title: Add an easy way to run pytest -Werror::DeprecationWarning -Werror::FutureWarning
body: CI is erroring on uncaught deprecation and future warnings but that's hard to reproduce locally unless you really know what you're doing.

I feel like we should make this easier, or maybe even the default when running tests locally?

One way would be to do this in the makefile, which would only help if using make, but that already would be better, I think.

We could also add it to the configuration, not sure if that's overkill?

It's a bit annoying to have tests pass locally but then not on CI because it uses different flags [yes yes, I added these flags, I know].
user: amueller
labels: Needs Decision - Close: Requires decision for closing
issue_13391: state: open
title: Add MultipleImputation
body: Now that IterativeImputation() has been added, it may be the right moment to discuss addition of yet another advanced imputation strategy: multiple imputation. The core idea is to aggregate across several imputed candidate values to better account for randomness.

Technical literature on multiple imputation includes Gelman & Hill 2007 and Harrell's RMS.
user: banilo
issue_13351: state: open
title: Pandas DataFrame Categories supported by OneHotEncoder
body: If one of the features is Pandas Categorical it is supported by OneHotEncoder and the categories will be kept as set

TODO:
- [ ] Validate the dtype between fit and transform (for now only allow exactly the same dtype)
- [ ] Deprecation path:
  - [ ] By default still do it the old way, but provide a way to opt in to the new behaviour
- [ ] Test that the order of the categories is used
user: maikia
issue_13339: state: open
title: Bad error messages in ClassifierChain on multioutput multiclass
body: trying to run classifier chain on a multiclass problem with a reshaped y (which is interpreted as multioutput multiclass) doesn't work (see #9245), which is fine. But the error messages are really bad making it hard to understand what's going on.

We should detect that we're trying to do multioutput multiclass instead of multilabel and give an informative error.
user: amueller
labels: good first issue: Easy with clear instructions to resolve
issue_13338: state: open
title: need "multilabel_only" tag.
body: Also see #9245
Right now ClassifierChain doesn't support multilabel multi-class which the common tests try to do.
There's currently "multioutput_only" but there's no "multilabel only". Should we just add "multilabel_only" and not have "multioutput_only" in this one?
user: amueller
labels: Needs Decision: Requires decision
issue_13330: state: open
title: Nearest neighbors with trees perf decreased by debugging stats
body: #### Description

For `ball_tree` and `kd_tree` algorithms, some stats about the tree queries highly decrease the parallelization performances increase.

Those stats are:

- `n_trims`: queried points outside node radius
- `n_leaves`: leaves reached while querying
- `n_splits`: non-leaves queried nodes
- `n_calls`: num of computed distances

Those stats only seem useful for debugging, do not look like part of the official API (no documentation) and only 2 (personal) git repos use the method (`get_tree_stats`) to get them.

**Deactivating them highly improves performances of associated algorithms.**

#### Benchmark

Test of kneighbors function with default parameters and:

- samples dimension: 100
- fit: 10k samples
- kneighbors: 10k samples

_(also tested openMP prange parallism but it does not improve perf)_

```
=============
=== brute ===
=============
Joblib (loky) :
- n_jobs = 1 (MKL mono threaded) -> 2.6s
- n_jobs = 1 (MKL multi threaded, 40 threads) -> 1.9s
- n_jobs = 4  -> 4.0s
- n_jobs = 10 -> 3.5s
- n_jobs = 40 -> 3.5s

=================
=== ball_tree ===
=================
Joblib (loky) :
- n_jobs = 1  -> 10.9s
- n_jobs = 4  ->  7.7s
- n_jobs = 10 ->  6.8s
- n_jobs = 40 ->  3.8s

Joblib (loky) no stats:
- n_jobs = 1  -> 12.0s
- n_jobs = 4  ->  3.2s
- n_jobs = 10 ->  1.4s
- n_jobs = 40 ->  0.6s

OpenMP no stats:
- n_jobs = 4  ->  3.2s
- n_jobs = 10 ->  1.4s

===============
=== kd_tree ===
===============
Joblib (loky) :
- n_jobs = 1  -> 19.1s
- n_jobs = 4  ->  9.0s
- n_jobs = 10 ->  10.9s
- n_jobs = 40 ->  8.5s

Joblib (loky) no stats:
- n_jobs = 1  -> 19.0s
- n_jobs = 4  ->  5.1s
- n_jobs = 10 ->  2.2s
- n_jobs = 40 ->  1.0s

OpenMP no stats:
- n_jobs = 4  ->  5.1s
- n_jobs = 10 ->  2.2s
```

user: rmenuet
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_13317: state: open
title: [WIP] Change default convergence params for classes using liblinear (LinearSVM and LogisticRegression)
body: #### Reference Issues/PRs
Fixes #11536 

#### What does this implement/fix? Explain your changes.
This Pull request is meant to implement an automatic determination of the value of convergence parameters (max_iter and tol).

#### TODO

- [x] Initialize the structure of the code
- [x] Fill in the blank left in the code structure to implement the switch on the solver type to determine max_iter and tol
- [x] Set defaults corresponding to solver defaults
- [ ] Check if solver defaults generates convergence warnings
- [ ] Identify better defaults through benchmarks
- [ ] Investigate changing the type of tol of lbfgs to ftol rather than pgtol
user: FollowKenny
labels: Needs Decision: Requires decision
issue_13288: state: open
title: Add example of creating a static transformer for transfer learning using joblib or onnx deserialisation
body: Often we want to transfer knowledge learnt from large-scale unlabelled data (e.g. decomposition, dictionary learning, etc.) to a smaller-scale supervised learning problem. We want to include such a pre-trained transformation in a Pipeline within a grid search.

This is a key use case for a freezing API (#8370) but a simpler and possible-now solution involves serialising the pre-trained model, and implementing an estimator which loads the model in a pipeline.

```py
class StaticTransformer(TransformerMixin):
    def __init__(self, pickle_path):
        ....
    def fit(self, X, y, ...):
        self.model_ = joblib.load(self.pickle_path)
        return self
    def transform(self, X):
        return self.model_.transform(X)
```

We would like to see an example of this paradigm of transfer learning, perhaps using ONNX for serialisation, added to our example gallery.

Ping @xadupre
user: jnothman
issue_13286: state: open
title: IterativeImputer shouldn't just use l2 loss by default
body: @GaelVaroquaux points out that iterative imputation with a regularised least-squares model is more-or-less the same as using NMF for imputation. We should instead use RandomForestRegressor as the default regressor in IterativeImputer, at least if sample_posterior=False (or we can implement predict(return_std=True) on RandomForestRegressor!).
user: jnothman
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_13285: state: open
title: Should fitting with warm_start=True validate changes in parameters?
body: I feel like at the moment, `warm_start=True` is hard to use because it is only relevant for some changes to parameters. Maybe we should consider storing the current params when `fit` is called if `warm_start=True`, and checking that the subsequent parameters can be warm started when `fit` is called again.
user: jnothman
issue_13269: state: open
title: [WIP] Resamplers
body: #### Reference Issues/PRs
Closes #4143
Closes #9630
Related SLEP: 
https://github.com/scikit-learn/enhancement_proposals/pull/15
#### What does this implement/fix? Explain your changes.
Implement estimators that can change the sample size at fit

#### Other comments
How could resamplers work with ColumnTransformer or FeatureUnion? 
Does the API allow the modification of `y` values? If so, should we reimplement TransformedTargetRegressor under this API? 

#### Plan
- [x] handle kwargs behavior
- [x] Rewrite all relevant common_tests involving fit to test fit_resample (see common test failures for FilterNaN)
- [x] FilterNaN Tests
- [ ] More estimator checks for resamplers
- [ ] More ResampledTrainer tests (including with FilterNaN and outlier rejectors)
- [ ] Docstrings for ResampledTrainer
- [ ] Port RandomOverSampler and RandomUnderSampler from imblearn 
- [ ] Sphinx Example(s)

user: orausch
labels: Needs Decision: Requires decision
issue_13246: state: open
title: [WIP] Common test for equivalence between sparse and dense matrices.
body: Fixes #1572 
Follow-up of #7590 
TODO:
- [x] <strike>Try to be more robust by maybe testing only the score (if possible), with a certain tolerance, and for certain random seeds (or if testing the predict also maybe test that like 90% of predictions are the same ?)</strike> 
After discussion with @agramfort , it would be better to test on a realistic dataset which would avoid weird instable behaviours
- [x] address https://github.com/scikit-learn/scikit-learn/pull/13246#pullrequestreview-208581402
- [x] Remove the fix for the Affinity propagation and put it in another PR
- [ ] Wait for the tests to pass thanks to other PR's fix (see for instance<strike> #13336 </strike> #17554 )
- [x] Replace the sparse format generator in ``check_estimator_sparse_dense`` by the one (more exhaustive) in ``check_estimator_sparse_data``
- [x] Address @jnothman's comment https://github.com/scikit-learn/scikit-learn/pull/13246#discussion_r532265287, and be more exhaustive in the fake dataset creation, in particular ensuring it has some zeros (for the sparsity to actually do sth), and <strike> that it tests well multiclass and multioutputs estimators </strike> <b>->see if it's necessary here to check with multioutput sparse datasets</b>
- [ ] Look at sparse matrices which may contain duplicates (see one of the comments in the thread I think)
user: wdevazelhes
labels: module:test-suite: everything related to our tests
issue_13213: state: open
title: PERF: Replace range by prange when applicable
body: We have recently enabled OpenMP support in sklearn, so let's use it !

It would be worth to check if there are place in sklearn cython code where we can use a prange loop instead of a range loop for instance.
user: jeremiedbb
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_13211: state: open
title: API Inconsitency of predict and predict_proba in SVC
body: When using `SVC(probability=True)` ~~or `SVR(probability=True)`~~ the output of `predict_proba` will not necessarily be consistent with `predict`, in the sense that,
```py
np.argmax(self.predict_proba(X), axis=1) != self.predict(X)
```
this is documented [in the user guide](https://scikit-learn.org/stable/modules/svm.html#scores-and-probabilities),
> In addition, the probability estimates may be inconsistent with the scores, in the sense that the “argmax” of the scores may not be the argmax of the probabilities. (E.g., in binary classification, a sample may be labeled by predict as belonging to a class that has probability <½ according to predict_proba.) Platt’s method is also known to have theoretical issues.

IMO this is a violation of the API contract and should be fixed. 

This is being continuously reported as a bug e.g.  https://github.com/scikit-learn/scikit-learn/issues/4800  https://github.com/scikit-learn/scikit-learn/issues/12408 https://github.com/scikit-learn/scikit-learn/issues/12982 and a few stack overflow issues e.g. https://stackoverflow.com/a/17019830 

I encountered this in a project where detecting this discrepancy, evaluating the difference and deciding whether `predict` or `argmax(predict_proba` should be used in the end took some effort.

One pitfall is for instance to use `predict` to compute the accuracy, and then `predict_proba` for ROC AUC which can lead to somewhat problematic results if the predictions of these methods are not consistent.

Several approaches could be used to fix it,

1. Deprecate `probability=True` parameter in `SVC`, `NuSVC` estimator and suggest using `CalibratedClassifierCV(SVC(), cv=5)` instead. In my quick tests (on sparse data), the latter was actually faster and should yield comparable results that are also consistent between predict and `predict_proba`. Though more benchmarks may be needed.
   There may also be some variation in the results, as libsvm uses a generalization of Platt scaling in the multiclass case by Wu et al 2014 (cf [docs](https://scikit-learn.org/stable/modules/svm.html#scores-and-probabilities) that is not used in `CalibratedClassifierCV` as far as I understand?

One possibility could be to deprecate, but keep it to allow access to that functionality in libsvm. 

2. Compute `predict` as `argmax(predict_proba` when `probability=True`. This has the disadvantage of changing the results of predict depending on this input parameter.

3. Dig into libsvm to understand how it could be fixed there. 

user: rth
labels: Needs Decision: Requires decision
issue_13201: state: open
title: Add n_outputs_ param to MultiOutputRegressor and MultiOutputClassifier
body: #### Description
Estimators supporting Multioutput out of the box (e.g. `DecisionTreeClassifier`) have an `n_outputs_` parameter (which is 1 when used in the classic, single-output mode).
I've tried replacing such inherently Multioutput classifier with a single-output classifier wrapped with `MultiOutputClassifier` wrapper, and my flow failed due to lack of `n_outputs_` in the wrapper.
Should be pretty straightforward to have the wrapper emulate this.

#### Versions
scikit-learn 0.20.1
user: obarak
labels: Needs Decision: Requires decision
issue_13180: state: open
title: New recommended ways to write (in)equality assertions in tests
body: I've had a few reviews recommending the use of `np.testing.assert_allclose()` (or was it `assert np.allclose()`?) instead of `assert_array_almost_equal` but there doesn't seem to be a consensus (https://github.com/scikit-learn/scikit-learn/pull/13130#issuecomment-464520286).

Also the use of `assert_equal` seems to be "deprecated" in favor of `assert . == .` (https://github.com/scikit-learn/scikit-learn/pull/13174/files#r257524501)

So, let's all agree on how we should write tests? I'll update the dev guide with the new decisions
user: NicolasHug
labels: Needs Decision: Requires decision
module:test-suite: everything related to our tests
issue_13171: state: open
title: License for rendered docs repo
body: The [rendered docs repo](https://github.com/scikit-learn/scikit-learn.github.io) could use a license.txt/[COPYING](https://github.com/scikit-learn/scikit-learn/blob/master/COPYING) to make the licensing more explicit.
 
My particular use case:
I'm looking to upload some of the rendered graphs to Wikimedia Commons for use in Wikipedia articles on ML and need to reference license. While I could reference the license from the website footer or backtrack to the graphs' source, I happened to start at the docs repo. 

<img src="https://github.com/scikit-learn/scikit-learn.github.io/blob/master/0.20/_images/sphx_glr_plot_learning_curve_001.png?raw=true">

user: justinormont
labels: Needs Decision: Requires decision
issue_13136: state: open
title: Add score_params to Pipeline.score()
body: In Pipeline.score() the final step is to call `self.steps[-1][-1].score(Xt, y, **score_params)` but the signature of Pipeline.score() is `score(self, X, y=None, __sample_weight=None__)` meaning the only valid score_params that are passed to the last step estimator of the pipeline is { 'sample_weight': sample_weight}

I created a custom estimator with custom score_params but it's not working, fit() and predict() allow arbitrary parameters, ideally score would as well.

Instead of passing my custom parameters, the pipeline passes sample_weight (with the None default) and my estimator raises an exception as it doesn't expect that parameter.
user: david-waterworth
labels: Needs Decision: Requires decision
issue_13131: state: open
title: FIX #13117: Remove multiple output from LinearModelCV doc
body: #### Reference Issues/PRs
Fixes #13117 

#### What does this implement/fix? Explain your changes.
Fixes documentation of class `LinearModelCV()`, since it there is a check that
does not allow `y` to have more than 1 dimension (multiple outputs)

#### Any other comments?
I thinks this could be improved, since both `ElasticNet()` and `Lasso()` support multiple targets. I don't see a reason why `LinearModelCV()` could not accept multiple targets as well.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: albertotb
labels: Needs Decision: Requires decision
issue_13127: state: open
title: SparsePCA implementation based on sparse dictionary learning
body: #### Description
Hi developers,

There's a sklearn model called [SparsePCA](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.SparsePCA.html), but it doesn't seem to be SPCA at all. According to documentation [here](https://scikit-learn.org/stable/modules/decomposition.html#sparse-principal-components-analysis-sparsepca-and-minibatchsparsepca), it's based on a sparse dictionary learning paper ([this one](https://www.di.ens.fr/sierra/pdfs/icml09.pdf)). This seems like a naming mistake, and is especially odd because there's also a model called [DictionaryLearning](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.DictionaryLearning.html) that seems to optimize a very similar sparse coding objective.

I was looking for an open-source SPCA implementation, ideally based on the [original paper](https://web.stanford.edu/~hastie/Papers/spc_jcgs.pdf), and was confused to find out that the sklearn one is something else altogether. I think I'm not the only one, e.g. I found open bug #11512 where a user requests a feature based on the SPCA paper that doesn't make as much sense in the context of dictionary learning.

Are there plans to fix this inconsistency? Would there be interest from other users in having a SPCA implementation?

I ended up implementing the model myself so would be happy to help, if this seems important.

Thanks,
Ian

#### References
H. Zou, T. Hastie, R. Tibshirani: Sparse Principal Components Analysis, Journal of Computational and Graphical Statistics, 2006.

J. Mairal, F. Bach, J. Ponce, G. Sapiro: Online Dictionary Learning for Sparse Coding, ICML 2009.
user: iancovert
issue_13117: state: open
title: Possible error in documentation of ElasticNetCV()
body: The documentation for class `ElasticNetCV()` reads:

    y : array-like, shape (n_samples,) or (n_samples, n_targets)

However, when I try to pass a 2D array I get the error:

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-33-597417f0a72a> in <module>()
      2 
      3 enet = ElasticNetCV()
----> 4 enet.fit(X_train, y_train)

/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/coordinate_descent.py in fit(self, X, y)
   1091             if y.ndim > 1 and y.shape[1] > 1:
   1092                 raise ValueError("For multi-task outputs, use "
-> 1093                                  "MultiTask%sCV" % (model_str))
   1094             y = column_or_1d(y, warn=True)
   1095         else:

ValueError: For multi-task outputs, use MultiTaskElasticNetCV
```


user: albertotb
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_13113: state: open
title: [Feature] Normalized Precision-Recall Metrics Added
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs


#### What does this implement/fix? Explain your changes.
* **Precision and Recall Metrics are heavily influenced** by **class imbalance** in a binary classification. 
* By computing **minimum AUCPR, minimum Average Precision, minimum Precision** for a given class balance, users will be able to make better quantitative comparisons on how this **imbalance affects their precision-recall based metrics**, without changing anything about the learner. 
* To quote the paper in which this is from: "For example, **changing the skew** of a data set from 0.01 to 0.5 **increases the minimum AUCPR** by approximately 0.3. This leads to an automatic jump of **0.3 in AUCPR simply by changing the data set** and with absolutely no change to the learning algorithm"

My code comes from translating Theorems 1 to 3 to code from Unachievable Region in Precision-Recall Space (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3858955/)

* Added normalized_aucpr function
  * which computes the normalized average precision and AUCpr, with respect to the smallest average precision/AUCpr possible due to the skew.
  * There is also a random guess normalization strategy to compute how well a classifier does in Precision-Recall against a random guess prediction.
  * This statistic will help data scientists answer the question of how well their precision-recall ranking metrics will improve above the smallest average precision/AUCpr/random guess possible due to the class imbalance.

Willing to take any comments/feedback.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.
Thanks for contributing!
-->

user: TheXu
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_13106: state: open
title: Model trees (M5P and co)
body: #### Description

There is an implementation of the CART algorithm where the leafs are a kind of model (linear regression, svr, neural net). And the split is done thanks to a criterion that depends on the chosen model.

https://github.com/ankonzoid/LearningX/tree/master/advanced_ML/model_tree

(the readme is quite easy to understand)

It looks really similar to scikit-learn API. And it would be awesome to have it in scikit-learn directly.

Would that be possible to add this in scikit-learn? (and who would be interested? I will, but alone...)

#### Other references.

Ross J. Quinlan: Learning with Continuous Classes. In: 5th Australian Joint Conference on Artificial Intelligence, Singapore, 343-348, 1992.

Y. Wang, I. H. Witten: Induction of model trees for predicting continuous classes. In: Poster papers of the 9th European Conference on Machine Learning, 1997. 

user: juliensiebert
issue_13099: state: open
title: gamma hyperparameter misspecified on KernelPCA
body: There is a missespecification on `_get_kernel` method at `KernelPCA` class.

If you perform the `inverse_transform` method specifiying `gamma` as `None` or directly imputing the value `1 / X.shape[1]`, it will differ, as the `_get_kernel` function with `gamma` as `None` adjusts to the `X` data both in `transform` and at `inverse_transform` methods.

The consequence is that specifying the supposed  default hyperparameter leads to different results than leaving it as `None`. 

The proposal is to add a hyperparameter called `inverse_gamma` or something like that for disambiguation.
user: Guillermogsjc
issue_13075: state: open
title: NuSVC hangs in some occasions
body: #### Description
When fitting certain data to the NuSVC classifier, there is a probability of the function halting (i.e. it never returns).

#### Code to Reproduce
```python
from sklearn.svm import NuSVC
from numpy import array

m = NuSVC(coef0=10.0, gamma="auto", kernel="poly", nu=2/7, probability=True)
X = array([[0. ,        0. , 0.3 , 0. ,  0. , 1. ,        0.625 ,     0. , 0. ,     0.31865753, 0. , 0. , 0. ],
           [0. ,        0. , 0. ,  0. ,  0. , 0.33333333, 0.61538462, 0. , 0. ,     0.59230137, 1. , 0. , 1. ],
           [0. ,        0. , 0. ,  0. ,  0. , 0. ,        0. ,        0. , 0. ,     0.31778082, 1. , 0. , 0. ],
           [0.66666667, 0. , 0. ,  0.5 , 1. , 0.8 ,       0. ,        1. , 0.5625 , 0.36430137, 0. , 0. , 0. ],
           [0. ,        0. , 0. ,  0. ,  0. , 0. ,        0. ,        0. , 0. ,     0.33638356, 1. , 0. , 0. ],
           [0. ,        0. , 0.3 , 0. ,  0. , 1. ,        0.625 ,     0. , 0. ,     0.31843836, 0. , 0. , 0. ]])
y = array([False,  True,  True, False,  True, False])

for i in range(100):
    print(i)
    m.fit(X, y)
```
#### Actual Results
The loop never finishes and halts at some point (usually after very few iterations). ^C is unable to terminate the script and the interpreter must be terminated through other means (SIGTERM).

#### Versions
Linux-4.19.13-1-MANJARO-x86_64-with-arch-Manjaro-Linux
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56) 
[GCC 7.2.0]
NumPy 1.14.3
SciPy 1.1.0
Scikit-Learn 0.19.1

user: gatl
labels: Needs Decision: Requires decision
issue_13063: state: open
title: RFC/MAINT: refactor and common tests for estimator supporting string (object dtype)
body: Following #13046, we have redundant code in estimator which should support string dtype (object dtype). Those estimators should make similar validation and raise consistent errors and have (a) common test(s).

The estimators which are involved will be:

* SimpleImputer
* MissingIndicator
* OneHotEncoder
* OrdinalEncoder

I open this issue such that we can agree on the way to refactor this part.
user: glemaitre
labels: module:test-suite: everything related to our tests
issue_13062: state: open
title: Re-evaluate memory usage in CountVectorizer
body: In https://github.com/scikit-learn/scikit-learn/pull/7272 (that was a mix of different PRs aiming to optimize text vectorizers) among other things, the intermediary storage of `X.indices` (where `X` is the document term matrix) was moved from `array.array('i')` to `List[int]`.

There were benchmarks suggesting that overall after that PR, less peak memory was used to vectorize documents. However, now I don't understand how that is possible since,
 - one element of `array.array('i')` should be around 8x smaller than that of a Python int
 - storage of indices of the output sparse arrays should be in large part responsible for the overall memory footprint

I can't see any obvious mistakes in my benchmarks back than, but something doesn't make sense here.

Re-measuring peak memory usage of `CountVectorizer.fit_transform`, with and without the above change would be a good place to start.

Related discussion in https://github.com/scikit-learn/scikit-learn/issues/13045#issuecomment-458300128
user: rth
issue_13058: state: open
title: Incorrect calculations of homogeneity, completeness and v-measure
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
Calculations of homogeneity, completeness and v-measure are now based on the original paper of Rosenberg & Hirschberg 2007. However, while I was doing research on fuzzy clustering evaluation techniques, I found the following paper of Utt et al. 2014 (http://www.lrec-conf.org/proceedings/lrec2014/pdf/829_Paper.pdf) which explained in a footnote that the original definitions of homogeneity and completeness contain typos. They claim it was confirmed by Rosenberg himself via personal communications.

Definitions used: 

- homogeneity = 1 - H(C|K) / H(C)
- completeness = 1  - H(K|C) / H(K)

Corrected definitions:

- homogeneity = 1 - H(C|K) / H(C,K)
- completeness = 1  - H(K|C) / H(K,C) 

Furthermore, since the calculations are now based on the mutual information score, this wouldn't be correct anymore. Also, the statement in the documentation about it being the same as normalized mutual information with the metric set to 'arithmetic' would be false.

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->
from sklearn.metrics import homogeneity_completeness_v_measure

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
System:
    python: 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 13:35:33) [MSC v.1900 64 bit (AMD64)]
executable: C:\Users\dtuser\AppData\Local\Programs\Python\Python36\python.exe
   machine: Windows-7-6.1.7601-SP1
BLAS:
    macros: 
  lib_dirs: 
cblas_libs: cblas
Python deps:
       pip: 18.1
setuptools: 40.6.3
   sklearn: 0.20.1
     numpy: 1.15.4
     scipy: 1.1.0
    Cython: None
    pandas: 0.23.4

<!-- Thanks for contributing! -->

user: royklip
issue_13043: state: open
title: precision_recall_curve should raise UndefinedMetricWarning when there are no true samples
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
When `precision_recall_score` is calculated with no true samples, a `RuntimeWarning` due to division by 0 is raised and `nan` is returned.  However, when `recall_score` is calculated with no true labels, a `UndefinedMetricWarning` due to no true samples is raised.

For consistency, I think `precision_recall_score` should raise the same error and return 0.

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->
```python
from sklearn.metrics import precision_recall_curve, recall_score
print(precision_recall_curve([0, 0], [0, 1]))
print(recall_score([0, 0], [0, 1]))
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
```
/Users/.../scikit-learn/sklearn/metrics/classification.py:1293: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 due to no true samples.
(array([0., 1.]), array([0.,  0.]), array([1]))
/Users/.../scikit-learn/sklearn/metrics/classification.py:1293: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 due to no true samples.
  'recall', 'true', average, warn_for)
0.0
```

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
```
/Users/.../scikit-learn/sklearn/metrics/ranking.py:601: RuntimeWarning: invalid value encountered in true_divide
  recall = tps / tps[-1]
(array([0., 1.]), array([nan,  0.]), array([1]))
/Users/.../scikit-learn/sklearn/metrics/classification.py:1293: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 due to no true samples.
  'recall', 'true', average, warn_for)
0.0
```

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
```
System:
    python: 3.7.2 (default, Dec 29 2018, 00:00:04)  [Clang 4.0.1 (tags/RELEASE_401/final)]
executable: /Users/.../anaconda3/envs/sklearn-py37/bin/python
   machine: Darwin-18.2.0-x86_64-i386-64bit

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: /Users/.../anaconda3/envs/sklearn-py37/lib
cblas_libs: mkl_rt, pthread

Python deps:
       pip: 18.1
setuptools: 40.6.3
   sklearn: 0.21.dev0
     numpy: 1.15.4
     scipy: 1.1.0
    Cython: 0.29.2
    pandas: 0.23.4
```

<!-- Thanks for contributing! -->

user: zjpoh
issue_13031: state: open
title: Machine readable definition of parameters
body: Related to #5004.
For automl purposes and automatically using scikit-learn more programmatically, it would be helpful for some to specify allowed parameters in a machine-readable format. Right now they are only available via the documentation, which doesn't allow us to actually test them and might be hard to parse.

Considerations are how to specify hyper-parameters. We could use the Python type system, but I think we would need to define custom types for strings with a fixed number of options.

It might also make sense to have possible ranges for integer and float parameters.

Finally, it might be interesting to annotate whether a parameter influences algorithm results, algorithm speed and/or memory consumption.

Not all options influence the result and I've seen people grid-search optimization options while evaluating only on accuracy. There might be fuzzy cases, though, like different solvers might yield slightly different results but it's mostly a time/memory issue.

Another question is whether it's in scope of doing this inside scikit-learn at all, or whether it should live outside - but it will be hard to keep in sync.

Being able to test for different options (not in standard CI but maybe in cron?) might be a good motivation. However, only some combinations of parameters are valid and actually specifying valid combinations (which would be required for testing) is more tricky than just specifying valid values.
user: amueller
issue_13025: state: open
title: [WIP] Example of multiple imputation with IterativeImputer
body: Adding to #11977. This PR is a restart of #11370, which got messy.

Here is a quote from #11370 that explains what this PR does:

> This PR is an example that shows how to use IterativeImputer for Multiple Imputation.
> 
> As discussed in #11259, the defaults of IterativeImputer are such that single imputation is performed. Because the method is also quite powerful for Multiple Imputation, we agreed to make an example that shows the user how to use ImputerImputer to perform Multiple Imputation.
> 
> I made the document: examples/impute/plot_multiple_imputation.py and it shows 2 things:
> 
> Estimation of beta estimates and their standard error: compare IterativeImputer with using IterativeImputer as a MICE Imputer.
> How to use IterativeImputer as a MICE Imputer when making a prediction model (with train and test datasets).
user: sergeyf
issue_13012: state: open
title: Improving non-interactive Usage
body: #### Description

Using scikit-learn in non-interactive mode (e.g. cronjob) needs to account for several corner cases among which are:

- classification of an empty sample set
- classification of sample sets with only 1 class (or less than n if estimator supports only more or equal n)
- classification of cross-validation with too few classes

Why? Because cronjobs usually aren't intelligent enough to neither handcraft unpleasant data nor using a different algorithm from time to time. A working classifier is needed as long as it can tell us about the confidence of a classification.

Why having a solution in scikit-learn? Because there are relevant SO questions out there without proper answers.

related: #13001, #6231, #8100, #12030

#### Actual Results
ValueError: Found array with 0 sample(s) (shape=(0, 3163)) while a minimum of 1 is required.
ValueError: This solver needs samples of at least 2 classes in the data, but the data contains only one class
ValueError: Requesting 3-fold cross-validation but provided less than 3 examples for at least one class.


#### Expected Results
Having a working classifier despite edge-casing data.


#### Preferred Solution
3 options, one for each case to handle them gracefully.

#### Current Solution
Problematic cases are try-excepted or if'ed away.

``` python
class PipelineWrapper(BaseEstimator):
    """
    Enables classification of an empty sample set.
    Enables classification of sample sets with only 1 class.
    Enables classification of cross-validation with too few classes.
    """

    def __init__(self, base_pipeline):
        self.base_pipeline = base_pipeline

    def fit(self, X, y, **fit_params):
        try:
            return self.base_pipeline.fit(X, y, **fit_params)
        except ValueError as exc:
            if 'This solver needs samples of at least 2 classes in the data' in exc.message:
                return
            elif 'cross-validation but provided less than' in exc.message:
                final_estimator = self.base_pipeline._final_estimator
                small_classes = {class_ for class_ in final_estimator.classes_ if np.sum(y == class_) < final_estimator.cv}
                X, y = zip(*((x, y_) for x, y_ in zip(X, y) if y_ not in small_classes))
                return self.base_pipeline.fit(X, y, **fit_params)
            raise

    def predict_proba(self, X):
        if not X:
            return np.zeros((0, 1))
        if len(self.classes_) == 1:
            return np.ones((len(X), 1))
        return self.base_pipeline.predict_proba(X)

    def predict(self, X):
        if not X:
            return np.zeros((0,))
        if len(self.classes_) == 1:
            return np.full_like(X, self.classes_[0])
        return self.base_pipeline.predict(X)

    @property
    def classes_(self):
        return self.base_pipeline._final_estimator.classes_
```
user: srkunze
issue_13008: state: open
title: ColumnTransformer with numpy structured arrays
body: Similar to #12545, but maybe easier to implement.
user: marco-c
issue_12990: state: open
title: Smarter BaseSearchCV memory management
body: In a search made to optimize a `Pipeline` the pipeline is copied with a **deep** copy. The deep copy has the problem that can copy multiple times things that should be shared between two objects. This problem happens during a scikit-learn search if you want to share information contained in one transformer to another. E.g.
```
class StandardizeTransformer():
    def fit(self, X):
        self.divisor = X.max()
    def transform(self,X):
        return X / self.divisor
    def anti_transform(self,X):
        return X * self.divisor

class AntiTransformer():
    def __init__(self, transformer):
        self.transfromer = transformer
    def transform(self,X):
        return self.transformer.anti_transform(X)
st = StandardizeTransformer()
pipe = Pipeline[('standardize', st), ... , ('destandardize', AntiTransformer(st))]
gs = GridSearchCv(pipe,...)
```

In this case, during the `GridSearchCv`, `st` is copied two times in two different address. This result in an unfitted `st` into the `AntiTransformer`

Solving this problem would add much more possibilities to scikit-learn e.g.
- destandardized metrics
-  possibility of pre-train and fine-tune approaches in a grid search
and much more

I suppose that this can be solved by implementing in the right way the `__deepcopy__()` in the BaseSearchCV as mentioned in the python's copy documentation:

> The difference between shallow and deep copying is only relevant for compound objects (objects that contain other objects, like lists or class instances):
 A shallow copy constructs a new compound object and then (to the extent possible) inserts references into it to the objects found in the original.
 A deep copy constructs a new compound object and then, recursively, inserts copies into it of the objects found in the original.
Two problems often exist with deep copy operations that don’t exist with shallow copy operations:
 Recursive objects (compound objects that, directly or indirectly, contain a reference to themselves) may cause a recursive loop.
 Because deep copy copies everything it may copy too much, such as data which is intended to be shared between copies.
user: thomasbridi
issue_12985: state: open
title: make_classification does not respect class weights as well as possible?
body: #### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
make_classification does not respect class weights as well as possible?

#### Steps/Code to Reproduce

```python
from sklearn.datasets import make_classification
import numpy as np

n_samples = 30
weights = [0.1, 0.2]
n_classes = 3
features, labels = make_classification(n_samples=n_samples, n_features=10, n_informative=3, n_classes=n_classes, weights=weights, flip_y=0)
unique, counts = np.unique(labels, return_counts=True)
# to look at which class has how many samples
class_to_counts = dict(zip(unique, counts))
print class_to_counts
# adding inferred weight to weights for assertion
if len(weights) < n_classes:
    weights.append(1. - sum(weights))
# This fails even though 30 is divisible by all (also the inferred) weights and flip_y = 0
assert(set(counts) == set([w*n_samples for w in weights]))
```

#### Expected Results
The assertion should not fail: 
If the division of n_samples by each weight results in an integer (in terms of math, not type), said integers are expected to be the actual number of samples in each respective class.
Therefore, class_to_counts should be
```
{0: 3, 1: 6, 2: 21}
```
#### Actual Results
Assertion fails. class_to_counts is
```
{0: 3, 1: 7, 2: 20}
```
#### Versions
```python
~/.local/lib/python2.7/site-packages/numpy/distutils/system_info.py:625: UserWarning: 
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  self.calc_info()
~/.local/lib/python2.7/site-packages/numpy/distutils/system_info.py:625: UserWarning: 
    Blas (http://www.netlib.org/blas/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [blas]) or by setting
    the BLAS environment variable.
  self.calc_info()
~/.local/lib/python2.7/site-packages/numpy/distutils/system_info.py:625: UserWarning: 
    Blas (http://www.netlib.org/blas/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [blas_src]) or by setting
    the BLAS_SRC environment variable.
  self.calc_info()

System
------
    python: 2.7.12 (default, Dec  4 2017, 14:50:18)  [GCC 5.4.0 20160609]
   machine: Linux-4.4.0-116-generic-x86_64-with-Ubuntu-16.04-xenial
executable: /usr/bin/python

BLAS
----
    macros: 
cblas_libs: cblas
  lib_dirs: 

Python deps
-----------
    Cython: 0.28.5
     scipy: 1.1.0
setuptools: 20.7.0
       pip: 18.1
     numpy: 1.15.2
    pandas: 0.23.4
   sklearn: 0.20.0
```
Please let me know if information on the BLAS or ATLAS is needed.
user: c-bg
labels: Needs Decision: Requires decision
issue_12967: state: open
title: TST check_transformers_unfitted not run on some transformers
body: Currently, there seem to be no common tests that ensure that calling `transform` of estimators without a fit raises a `NotFitedError`. 

One instance of this issue is https://github.com/scikit-learn/scikit-learn/issues/12959

There is a common test for `predict` and some other methods in [`sklearn/utils/estimator_checks.py:check_estimators_unfitted`](https://github.com/scikit-learn/scikit-learn/blob/2955129128341ba3f2187c5a6619084f312bca88/sklearn/utils/estimator_checks.py#L1586). It might be worthwhile to extend that also to the transform method (if it is present).  

Though we probably want to enforce this only to a subset of estimators (maybe estimator tags) would have been useful here.
user: rth
labels: module:test-suite: everything related to our tests
issue_12960: state: open
title: RadiusNeighborsRegression is inconsistent when extrapolation occurs
body: #### Description
The behavior of `RadiusNeighborsRegression` is inconsistent when extrapolation occurs. The behavior depends on the chosen weight-function. 

-`weight="uniform"` will return `[NaN]`.
-`weight="distance"`will raise an error
-`weight=lambda d: d`will raise an error


#### Steps/Code to Reproduce
```python
from sklearn.neighbors import RadiusNeighborsRegressor

X = [[1],[2],[3],[4]]
y = [1,2,3,4]

X_predict = [[-100]]

model = RadiusNeighborsRegressor(radius=1.0, weights = "distance")
fitm = model.fit(X,y)

# raises ZeroDivisionError
result = fitm.predict(X_predict)  
```

#### Expected Results
No error is raised and `[[NaN]]` is returned. 

#### Actual Results
```
 File "c:/test.py", line 12, in <module>
    result = fitm.predict(X_predict)
  File "C:\...\anaconda3\lib\site-packages\sklearn\neighbors\regression.py", line 296, in predict
    for (i, ind) in enumerate(neigh_ind)])
  File "C:\...\anaconda3\lib\site-packages\sklearn\neighbors\regression.py", line 296, in <listcomp>
    for (i, ind) in enumerate(neigh_ind)])
  File "C:\...\anaconda3\lib\site-packages\numpy\lib\function_base.py", line 1158, in average
    "Weights sum to zero, can't be normalized")
ZeroDivisionError: Weights sum to zero, can't be normalized
```

#### Versions
```
Windows-8.1-6.3.9600-SP0
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]
NumPy 1.14.3
SciPy 1.1.0
Scikit-Learn 0.19.1
```
user: ErikDeSmedt
issue_12957: state: open
title: Naive Bayes Classifier with Mixed Bernoulli/Gaussian Models
body: #### Description
I suggest allowing mixed datasets (half binary variables, half real-valued variables) into the Naive Bayes classifier. Currently the `GaussianNB` and `BernoulliNB` classes handle one case or the other, but not combined. I'd be happy to write the code for this, so I'm curious if this has been explored before and if it would be helpful!

For example, on the Titanic dataset, gender is a bernoulli variable while age is real-valued. Passing both into a Naive Bayes classifier would improve it.

This is related to this currently pending PR: https://github.com/scikit-learn/scikit-learn/pull/12569
user: jarednielsen
issue_12956: state: open
title: unable to filterwarnings when using parallel RandomizedSearchCV
body: #### Description

`warnings.filterwarnings` doesn't seem to work with `RandomizedSearchCV` with `n_jobs=-1`. I have set:

```python
import warnings
warnings.filterwarnings('ignore', message=r'Label not .* is present in all training examples.')
warnings.filterwarnings('ignore', message=r'F-score is ill-defined and being set to 0.0 due to no predicted samples.')
```

This is my example usage:

```python
scv = RandomizedSearchCV(model, param_distributions, n_iter=n_iter, scoring=scoring, cv=cv, iid=True, verbose=2, n_jobs=-1)
```

When `n_jobs` is not set at all, then `filterwarnings` works totally fine. But when it is set, the warnings no longer get filtered. I tested by checking the behavior from `ignore` to `error` or even removing the regex and just setting `category=UserWarning` but nothing worked.

Here are some sample warnings:

```
running parameter search for alpha
Fitting 3 folds for each of 10 candidates, totalling 30 fits
[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
[CV] estimator__alpha=1.1983275213240523e-05 .........................
[CV] estimator__alpha=1.90571852162917e-05 ...........................
[CV] estimator__alpha=1.1983275213240523e-05 .........................
[CV] estimator__alpha=1.1983275213240523e-05 .........................
[CV] estimator__alpha=9.092657871686527e-06 ..........................
[CV] estimator__alpha=1.90571852162917e-05 ...........................
[CV] estimator__alpha=9.092657871686527e-06 ..........................
[CV] estimator__alpha=1.90571852162917e-05 ...........................
/Users/atul/Development/taxonomy-analysis2/.venv/lib/python3.7/site-packages/sklearn/multiclass.py:76: UserWarning: Label not 25 is present in all training examples.
  str(classes[c]))
/Users/atul/Development/taxonomy-analysis2/.venv/lib/python3.7/site-packages/sklearn/multiclass.py:76: UserWarning: Label not 25 is present in all training examples.
  str(classes[c]))
/Users/atul/Development/taxonomy-analysis2/.venv/lib/python3.7/site-packages/sklearn/multiclass.py:76: UserWarning: Label not 40 is present in all training examples.
  str(classes[c]))
...
...
...
[CV] ........... estimator__alpha=5.146393036919477e-06, total= 3.4min
/Users/atul/Development/taxonomy-analysis2/.venv/lib/python3.7/site-packages/sklearn/multiclass.py:76: UserWarning: Label not 87 is present in all training examples.
  str(classes[c]))
[CV] ........... estimator__alpha=5.146393036919477e-06, total= 2.5min
[Parallel(n_jobs=-1)]: Done  30 out of  30 | elapsed: 13.5min finished
found alpha 6.399242127649889e-06 with f1 score of 0.7866834499268398
```

#### Versions
```
System:
    python: 3.7.2 (default, Dec 27 2018, 07:35:06)  [Clang 10.0.0 (clang-1000.11.45.5)]
executable: /Users/atul/Development/taxonomy-analysis2/.venv/bin/python
   machine: Darwin-18.2.0-x86_64-i386-64bit

BLAS:
    macros: NO_ATLAS_INFO=3, HAVE_CBLAS=None
  lib_dirs:
cblas_libs: cblas

Python deps:
       pip: 18.1
setuptools: 40.6.3
   sklearn: 0.20.2
     numpy: 1.16.0rc2
     scipy: 1.2.0
    Cython: None
    pandas: 0.23.4
```

I opened an issue on joblib as well but I'm not 100% sure where the fix is needed: https://github.com/joblib/joblib/issues/832

This [SO comment](https://stackoverflow.com/questions/32612180/eliminating-warnings-from-scikit-learn#comment88325949_33812427) by Reed Richards states that (emphasis mine):

>It seems that the filterwarnings call needs to be **in the file that calls the function that gives the warning**. Calling it at the top level does nothing for me but when I put it in the file that gives the warning it does. And it should be clarified that you need to import whatever warning sklearn is throwing from its defined set of exceptions. – Reed Richards Jun 2 '18 at 14:37

I'm not sure if this is relevant here.
user: AlJohri
issue_12952: state: open
title: Implementing partial_fit for NearestCentroid
body: #### Description
The NearestCentroid classifier (https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestCentroid.html) does not currently have a partial_fit method, but it would be very useful when running the algorithm in an online fashion.  The math for implementing it is simple; it's just a matter of updating the centroids weighted by the number of samples.  The implementation could be similar to the partial_fit method of the GaussianNB classifier (except for the obvious mathematical differences).  I'm happy to submit an implementation of this, but I wanted to gather input first to see if others have already done this and whether others would find this useful. 

#### Steps/Code to Reproduce
Look at NearestCentroid classifier code and notice that it has not partial_fit method.

#### Expected Results
There should be a partial_fit method.

#### Actual Results
There is no partial_fit method.

#### Versions
N/A
user: dirkpadfield
labels: Easy: Well-defined and straightforward way to resolve
issue_12947: state: open
title: Using pre-fitted GradientBoostingRegressor init estimator
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
Scikit-learn allows to put a base estimator for some models like gbm by using an init parameter. But base estimator is again fitted inside the parent gbm model irrespective of whether the base estimator is already fitted or not. If i want to fit a base estimator where the target is modified version of the target (for ex logarithm of target for regression) but parent gbm model should train on the actual target then I get erroneous values as it tries to refit the base estimator again. We need a flag in the wrapper to check if we want to refit it again. Otherwise we have to write the entire fit function in the init base estimator class
#### Steps/Code to Reproduce
from sklearn.utils import check_array


class GBR_Init(GradientBoostingRegressor):
    def predict(self,X):
        X = check_array(X, dtype=np.float32, order='C', accept_sparse='csr')
        return np.expm1(self._decision_function(X))
#here i am training base estimator with log of the target
gbm_base=GBR_Init(random_state=1,verbose=1)
gbm_base.fit(X_train,np.log1p(y_train))

gbm_nolog=GradientBoostingRegressor(init=gbm_base,random_state=1,verbose=1)
gbm_nolog.fit(X_train,y_train)

#### Expected Results
#somthing like below
Iter       Train Loss   Remaining Time 
         1           2.0273            0.40s
         2           1.8393            0.25s
         3           1.6864            0.19s
         4           1.5590            0.17s
         5           1.4532            0.13s
         6           1.3650            0.13s
         7           1.2913            0.12s
         8           1.2277            0.12s
         9           1.1754            0.11s
        10           1.1207            0.11s
        20           0.7387            0.06s
        30           0.5735            0.05s
        40           0.5004            0.04s
        50           0.4405            0.03s
        60           0.3838            0.02s
        70           0.3349            0.02s
        80           0.2955            0.01s
        90           0.2611            0.01s
       100           0.2389            0.00s
      Iter       Train Loss   Remaining Time 
         1    52988925.8347            0.00s
         2    51270724.5307            0.00s
         3    48524692.1194            0.03s
         4    46375604.2136            0.02s
         5    44055967.6390            0.04s
         6    42184621.7979            0.03s
         7    41030696.3810            0.04s
         8    39307064.6078            0.03s
         9    37784980.9644            0.04s
        10    36529113.4140            0.04s
        20    29294392.9067            0.03s
        30    23815067.3185            0.02s
        40    19853773.5911            0.02s
        50    17400325.0065            0.02s
        60    14903616.7271            0.01s
        70    13433238.2705            0.01s
        80    11575990.5895            0.01s
        90     8788539.5693            0.00s
       100     7758800.1583            0.00s
GradientBoostingRegressor(alpha=0.9, criterion='friedman_mse',
             init=GBR_Init(alpha=0.9, criterion='friedman_mse', init=None, learning_rate=0.1,
     loss='ls', max_depth=3, max_features=None, max_leaf_nodes=None,
     min_impurity_decrease=0.0, min_impurity_split=None,
     min_samples_leaf=1, min_samples_split=2, min_weight_fraction_leaf=0.0,
     n_estimators=100, n_iter_no_change=None, presort='auto',
     random_state=1, subsample=1.0, tol=0.0001, validation_fraction=0.1,
     verbose=1, warm_start=False),
             learning_rate=0.1, loss='ls', max_depth=3, max_features=None,
             max_leaf_nodes=None, min_impurity_decrease=0.0,
             min_impurity_split=None, min_samples_leaf=1,
             min_samples_split=2, min_weight_fraction_leaf=0.0,
             n_estimators=100, n_iter_no_change=None, presort='auto',
             random_state=1, subsample=1.0, tol=0.0001,
             validation_fraction=0.1, verbose=1, warm_start=False)

#### Actual Results
Iter       Train Loss   Remaining Time 
         1   184762966.1171            0.40s
         2   169913485.2189            0.34s
         3   156524538.5406            0.26s
         4   145658964.4509            0.22s
         5   136687743.4135            0.19s
         6   129237345.2838            0.17s
         7   123040960.6351            0.16s
         8   116277891.9883            0.15s
         9   110082715.5338            0.14s
        10   106297995.0147            0.13s
        20    76520331.4205            0.08s
        30    57637099.1301            0.06s
        40    47219572.9258            0.04s
        50    39908521.2757            0.03s
        60    33645465.4188            0.02s
        70    27952255.6279            0.02s
        80    24589782.5990            0.01s
        90    21409717.9802            0.01s
       100    19287548.1185            0.00s
      Iter       Train Loss   Remaining Time 
         1              nan            0.10s
         2              nan            0.10s
         3              nan            0.06s
         4              nan            0.07s
         5              nan            0.06s
         6              nan            0.06s
         7              nan            0.05s
         8              nan            0.06s
         9              nan            0.05s
        10              nan            0.05s
        20              nan            0.04s
        30              nan            0.03s
        40              nan            0.03s
        50              nan            0.02s
        60              nan            0.02s
        70              nan            0.01s
        80              nan            0.01s
        90              nan            0.01s
       100              nan            0.00s
C:\Anaconda3\lib\site-packages\ipykernel_launcher.py:123: RuntimeWarning: overflow encountered in expm1
C:\Anaconda3\lib\site-packages\sklearn\ensemble\gradient_boosting.py:501: RuntimeWarning: invalid value encountered in add
  y_pred[:, k] += learning_rate * tree.predict(X).ravel()
GradientBoostingRegressor(alpha=0.9, criterion='friedman_mse',
             init=GBR_Init(alpha=0.9, criterion='friedman_mse', init=None, learning_rate=0.1,
     loss='ls', max_depth=3, max_features=None, max_leaf_nodes=None,
     min_impurity_decrease=0.0, min_impurity_split=None,
     min_samples_leaf=1, min_samples_split=2, min_weight_fraction_leaf=0.0,
     n_estimators=100, n_iter_no_change=None, presort='auto',
     random_state=1, subsample=1.0, tol=0.0001, validation_fraction=0.1,
     verbose=1, warm_start=False),
             learning_rate=0.1, loss='ls', max_depth=3, max_features=None,
             max_leaf_nodes=None, min_impurity_decrease=0.0,
             min_impurity_split=None, min_samples_leaf=1,
             min_samples_split=2, min_weight_fraction_leaf=0.0,
             n_estimators=100, n_iter_no_change=None, presort='auto',
             random_state=1, subsample=1.0, tol=0.0001,
             validation_fraction=0.1, verbose=1, warm_start=False)

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->


<!-- Thanks for contributing! -->

user: iamdbestjoy
issue_12945: state: open
title: Add a backward compatible flag `train_score_size` to the cross_validate function.
body: 
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
I did not create a separate issue since this is a small code change.

#### What does this implement/fix? Explain your changes.
This code adds a backward compatible flag `train_score_size` to the cross_validate function.
With this flag, we can request to subsample the training set for computing the train_scores
to save time. Currently, the documentation just warns users that setting `return_train_score` 
to True may be time-consuming and slow. This flag offers a way to improve this situation.


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: se4u
labels: Needs Decision: Requires decision
issue_12943: state: open
title: ENH Add new smoothing methods to MultinomialNB
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Resolves #12862 

#### What does this implement/fix? Explain your changes.

Implements the Good-Turing smoothing algorithm for Naive Bayes classifier and adds two other possible options.

#### Any other comments?

If the maintainers decide to add Jelinek-Mercer or Absolute Discounting, we need to use different default smoothing parameters depending on the selected algorithm. 

Since Good-Turing uses raw counts, the code won't work if the input is transformed using, for example, [tf-idf](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html). Would throwing an error be a reasonable solution when the input is not raw counts?

At first, I implemented Simple Good-Turing operating on the entire matrix `self.feature_count_` but I the current solution is more readable. For more information on the notation see [Good-Turing Frequency Estimation Without Tears](https://www.grsampson.net/AGtf1.html).

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: psendyk
issue_12939: state: open
title: RandomizedSearchCV: warnings cannot be suppressed
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->

I would like to suppress the warnings thrown by sklearn. This can be done while I was working on a single classifier, however, it failed on SearchCV task. I am wondering if this is designed for purpose and if there are any workarounds.

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

**File 1 (No warnings)**
```python
import warnings
import sklearn
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RandomizedSearchCV
import pandas as pd
import numpy as np


x = np.random.normal(0, 1, (100, 100))
y = [0] * 50 + [1] * 50

if __name__ == "__main__":
    warnings.filterwarnings("ignore")
    clf = LogisticRegression()
    clf.fit(x, y)
    clf.predict(x)
    print("done")
```
> Run the code from terminal

**File 2 (Warnings still exist)**
```python
import warnings
import sklearn
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RandomizedSearchCV
import pandas as pd
import numpy as np


x = np.random.normal(0, 1, (100, 100))
y = [0] * 50 + [1] * 50

if __name__ == "__main__":
    warnings.filterwarnings("ignore")
    clf = LogisticRegression()
    clf = RandomizedSearchCV(clf, {"C": [1,2]}, n_jobs=-1)
    clf.fit(x, y)
    clf.predict(x)
    print("done")
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
should see `done` from terminal. While the second file gave me multiple `FutureWarning`

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
As above
#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
```
appnope==0.1.0
backcall==0.1.0
certifi==2018.11.29
Click==7.0
cycler==0.10.0
decorator==4.3.0
feather-format==0.4.0
ipykernel==5.1.0
ipython==7.2.0
ipython-genutils==0.2.0
jedi==0.13.2
jupyter-client==5.2.4
jupyter-core==4.4.0
kiwisolver==1.0.1
matplotlib==3.0.2
mkl-fft==1.0.6
mkl-random==1.0.2
numpy==1.15.4
pandas==0.23.4
parso==0.3.1
pexpect==4.6.0
pickleshare==0.7.5
prompt-toolkit==2.0.7
ptyprocess==0.6.0
pyarrow==0.11.1
Pygments==2.3.1
pyparsing==2.3.0
python-dateutil==2.7.5
pytz==2018.7
pyzmq==17.1.2
scikit-learn==0.20.2
scipy==1.1.0
six==1.12.0
tornado==5.1.1
traitlets==4.3.2
wcwidth==0.1.7
xlrd==1.2.0
```
Tested on 3.6.8 with MacOS 10.14

<!-- Thanks for contributing! -->

user: yihengli
issue_12922: state: open
title: Error while using n_jobs=-1 with uwsgi
body: Hi I am using sklearn n_jobs = -1 for multi process training of my model but i am getting below error.

sklearn.externals.joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker. The exit codes of the workers are {EXIT(1), EXIT(1), EXIT(1)}
user: manishprasadthapliyal
issue_12920: state: open
title: Support for n_jobs (joblib) in simpleimpute
body: Can SimpleImpute() support n_jobs or joblib parallelism? the imputation process is extremely slow on an average-sized dataset.
user: wmlba
labels: Moderate: Anything that requires some knowledge of conventions and best practices
Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_12898: state: open
title: [MRG] Bugfix for not-yet-confirmed issue #12863: arpack returns singular values in ascending order, the opposite was supposed in sklearn
body: Bugfix for not-yet-confirmed issue #12863: arpack returns singular values in ascending order, the opposite was supposed in sklearn

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
Fixes [MRG] not-yet-confirmed issue #12863 

<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.

In the sklearn code it was assumed that the scipy wrapper for arpack singular value decomposition returns singular values (and corresponding vectors) in descending order. This is not the case. The ARPACK documentation (https://www.caam.rice.edu/software/ARPACK/UG/node136.html#SECTION001210000000000000000
) clearly states that the singular values are returned in ascending order.

The bug manifests in differing results depending on the solver:

```
import numpy as np
from sklearn import cluster

A= np.array([[-2, -4, 2], [-2, 1, 2], [4, 2, 5]])

sc= cluster.bicluster.SpectralCoclustering(n_clusters= 2, 
                                           svd_method='randomized')

sc.fit(A)
print(sc.column_labels_)
```
gives
```
[0 0 1]
```
, but
```
sc= cluster.bicluster.SpectralCoclustering(n_clusters= 2, 
                                           svd_method='arpack')
sc.fit(A)
print(sc.column_labels_)
```
gives
```
[0 0 0]
```

This bugfix makes the arpack call return the same result as the randomized sv based solution.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: gykovacs
issue_12866: state: open
title:  NOCATS: Categorical splits for tree-based learners (ctnd.)
body: This PR continues the work of #4899. For now I've merged the master into the PR, made it compile and make the tests run. There are several issues which need to be fixed. The list will be updated as I encounter them. Also, not all of these items are necessarily open, I have only collected them from the comments on the original PR, and need to make sure they're either already addressed or address them.

- merge master into the PR (__done__)
- sparse tests pass (__done__)
    - The code is supposed to be the same as the status quo implementation if categories are not passed. But right now the tests related to sparse data fail.
    - __EDIT__: The tests pass if we compare floats with `almost_equal`
- LabelEncoder -> CategoricalEncoder (__done__)
    - Preprocessing is not a part of NOCATS anymore.
- Is maximum random generations 20 or 40 (__done__)
    - It's actually 60
- Don't quantize features automatically (__done__)
    - Doesn't happen anymore: https://github.com/scikit-learn/scikit-learn/pull/4899#issuecomment-271504258
- check the category count limits for given data. (__done__)
- add a benchmark
   - __done__. Results: https://github.com/scikit-learn/scikit-learn/pull/12866#issuecomment-453856876
- add tests (right now only invalid input are tested)
   - `tree/tests` __done__
   - `ensemble/tests` __done__
- benchmark against master
- add an example with plots
- check numpy upgrade related issues (we've upgraded our numpy requirement in the meantime)
- run some benchmarks with a simple integer coding of the features (with arbitrary ordering)
- add cat_split to NODE_DTYPE once joblib.hash can handle it (padded struct)
    - joblib issue: joblib/joblib#826

Closes #4899

__Future Work__: These are the possible future work we already know of (i.e. outside the scope of this PR):

- Heuristic methods to allow fast Breiman-like training for multi-class classification
- export to graphviz
- One-hot emulation using the NOCATS machinery
- support sparse input
- handle categories as their unique valies instead of `[0, max(feature)]`
  - This is to be consistent with our encoders' behavior
  - moved this to future work per https://github.com/scikit-learn/scikit-learn/pull/12866#issuecomment-455021204


P.S. I moved away from "task list" due to the extremely buggy interface when used in combination with editing the post, which I'm extensively doing to keep it easy for us to keep up with the status.
user: adrinjalali
issue_12863: state: open
title: sklearn.cluster.bicluster.BaseSpectral._svd: n_discard eigenvectors from svds
body: #### Description

The function sklearn.cluster.bicluster.BaseSpectral._svd incorrectly uses the parameters svd_method = 'arpack' and n_discard.

The function _svd should discard the eigenvectors with largest eigenvalues, but the function svds used when svd_method=='arpack' returns the eigenvectors with ascending eigenvalues. This behavior is different with that of the function randomized_svd used when svd_method=='randomized'.
user: chuanren
issue_12862: state: open
title: Can Naive bayes in sklearn support good-turing smoothing?
body: #### Description
Naive bayes in sklearn seems do not support additive smoothing only.
Can we make some change to make it support good-turing smoothing or other smoothing and how?
In some experiment, good-turing smoothing is better than additive smoothing.

user: 12ycli
issue_12849: state: open
title: [WIP]  Make SVC tests independent of SV ordering 
body: This PR changes the assertions made on the parametrization of the dual coefficients of SVM to check the equality of two models because the ordering of the support vectors is non-deterministic in case of duplicated samples in the training set (as is the case in the iris dataset and the subset of 20 newsgroups
 we use in some of the tests).

The problem was identified in #12738 when trying to replace the libsvm-base solver by the one from DAAL using the daal4py monkeypatch from https://github.com/IntelPython/daal4py/pull/15.

I have tried to run the tests with:

```
python -m daal4py -m pytest -vl  sklearn/svm/tests
```

and they pass on this branch (using the daal4py branch from  https://github.com/IntelPython/daal4py/pull/15).

However, if I decrease the regularization (larger C value) in `test_svc_iris` or in `test_sparse_20newsgroups_subset` they will fail. I am not sure whether this reveals a true discrepancy between the 2 solvers or  whether this is just that the optimization problem becomes too under determined to guarantee such a strict equivalence of the estimated parametrization of the decision function.


user: ogrisel
issue_12845: state: open
title: LogisticRegressionCV failure when not all classes appear in each fold
body: #### Description

When not all classes appear in each CV fold, LogisticRegressionCV fails when trying to collect the coefficient array.

#### Steps/Code to Reproduce
```python
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegressionCV

xtmp, ytmp = make_classification(n_samples=2500, n_informative=10, n_classes=6)

# remove all but 2 instances of class 3, in order to make class 3 not-appear in at least one fold
remove = np.where(ytmp == 3)[0][2:]
keep = np.setdiff1d(np.arange(len(ytmp), dtype='int'), remove)

lr = LogisticRegressionCV(multi_class='multinomial', cv=KFold(5)).fit(xtmp[keep], ytmp[keep])
```

#### Expected Results
No errors; `lr` contains a fitted instance of `LogisticRegressionCV`.

#### Actual Results
```ValueError                                Traceback (most recent call last)
<ipython-input-51-ac4dd6663338> in <module>
      4 remove = np.where(ytmp == 3)[0][2:]  # remove all but 2 instances of class 3
      5 keep = np.setdiff1d(np.arange(len(ytmp), dtype='int'), remove)
----> 6 lr = LogisticRegressionCV(multi_class='multinomial', cv=KFold(5)).fit(xtmp[keep], ytmp[keep])

<redacted>/lib/python3.6/site-packages/sklearn/linear_model/logistic.py in fit(self, X, y, sample_weight)
   1796         if multi_class == 'multinomial':
   1797             multi_coefs_paths, Cs, multi_scores, n_iter_ = zip(*fold_coefs_)
-> 1798             multi_coefs_paths = np.asarray(multi_coefs_paths)
   1799             multi_scores = np.asarray(multi_scores)
   1800 

<redacted>/lib/python3.6/site-packages/numpy/core/numeric.py in asarray(a, dtype, order)
    499 
    500     """
--> 501     return array(a, dtype, copy=False, order=order)
    502 
    503 

ValueError: could not broadcast input array from shape (10,5,21) into shape (10)
```

If you go inspect the `multi_coefs_paths` object, you'll see that its elements have different shapes, due to the misaligned classes. Only workaround as far as I can tell is to do grid search manually.

#### Versions

```
System:
    python: 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44)  [GCC 7.3.0]
executable: <redacted>/bin/python
   machine: Linux-3.10.0-514.26.2.el7.x86_64-x86_64-with-redhat-7.3-Maipo

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: <redacted>/lib
cblas_libs: mkl_rt, pthread

Python deps:
       pip: 18.1
setuptools: 40.6.2
   sklearn: 0.20.1
     numpy: 1.15.4
     scipy: 1.1.0
    Cython: 0.29
    pandas: 0.23.4
```
user: gwerbin
issue_12841: state: open
title: [MRG] Implement randomized PCA
body: #### Reference Issues/PRs
Following the discussion in #12794, I've changed the randomized SVD implementation so it now supports computing PCA as well, without ever having to explicitly centering X. This is especially useful for large sparse matrices, which can't be centered.


#### What does this implement/fix? Explain your changes.

1. Add degrees of freedom (ddof) parameters to sparse functions that compute variance. Error checking has been added for negative ddof, when `ddof >= n_samples` or when the data contain NaNs and `n_samples - ddof - num_nans <= 0`.

2. Implement the randomized PCA algorithm. I was unable to reuse the `randomized_svd` (although the algorithms are conceptually very similar, their code is slightly different).

    In the case where there are more features than samples, it is more efficient to transpose the matrix and work with that. However, simply transposing isn't ok, because we also need the means from the original columns. This would make `randomized_svd` quite complicated. Furthermore, `randomized_svd` is strongly tied to `randomized_range_finder`, and I would need to add similar changes to both functions, making them quite confusing. For these reasons, I strongly feel adding the similar `randomized_pca` function is justified.

#### Any other comments?

The previous behaviour raised a `TypeError` if a sparse matrix was passed to `PCA.fit`. The behaviour is now somewhat different. Sparse matrices are supported with `svd_solver='randomized'`, otherwise a `ValueError` is raised indicating the wrong solver. If the solver is set to `auto` then the `randomized` solver is always selected for sparse matrices.



user: pavlin-policar
labels: Needs Decision: Requires decision
issue_12839: state: open
title: New feature: add color argument to tree export_graphviz  (at least for classification)
body: The export_graphviz when applied on a classification tree can color the node according to the assigned class.

The color code used for that class is automatically built using the _color_brew function.

It would be nice to let the user gives these color as argument to the export_graphviz function.

A typical use case is when the user is already creating other graphs with color assigned to the class. It will be nice to have the same "color code convention" on all graphs.
user: bertrandhaut
issue_12794: state: open
title: PCA on sparse, noncentered data
body: I suppose this is more of a feature request than anything else. There are several implementations of PCA that can compute the decomposition on noncentered, sparse data, while the implementation here does not support sparse matrices at all.

A matlab implementation can be found [here](https://github.com/KlugerLab/pcafast/blob/master/src/pcafast.m) and a Python implementation [here](https://github.com/facebook/fbpca). So far, I've been using the Python implementation, but it's missing some things and will eventually be deprecated (https://github.com/facebook/fbpca/pull/9).

I haven't looked at the code or math too much, but as far as I'm aware, it's just a matter of adding a term to `randomized_range_finder` to account for centering.

Is this something that you guys are aware of, and is anyone working on this? This would an awesome feature to have.
user: pavlin-policar
issue_12752: state: open
title: Is there a way in this package to implement semi supervised regression?
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description

I just checked and there is only "label propagation" under "supervised learning"
Is there any existing method regarding semi-supervised regression or at least a plan to release one in the future version?

<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->


<!-- Thanks for contributing! -->

user: yuanqingye
issue_12738: state: open
title: Plea to not use datasets with duplicate samples in SVC tests
body: #### Description
Support vector classification solves a quadratic programming problem, see [sklearn's user guide secion on SVC](https://scikit-learn.org/stable/modules/svm.html#svc). 

The quadratic form Q[i, j] is constructed from kernel function values at pairs (i, j) of input samples. 

Duplicate samples thus cause quadratic form to have zero eigenvalues. In such a case the solution of the quadratic problem is not unique. 

It is important for SVM code to handle such cases, and libSVM resorts to use of regularization parameter `tau` (see [src/libsvm/svm.cpp#L89](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/src/libsvm/svm.cpp#L89)). So any eigenvalues smaller than `tau` are replaced with `tau`, essentially resolving the non-uniqueness in a machine arithmetic fragile way.

A change in implementation may result in a different set of support vectors found.

Inputs with duplicates are pervasive in sklearn's test suite. 
Duplicates arise from use of grouped data, members of groups being represented by the same value. In iris dataset, the length and width of petals are only recorded with single decimal place precision, creating duplicates

```
In [17]: iris.data[[101, 142]]
Out[17]:
array([[5.8, 2.7, 5.1, 1.9],
       [5.8, 2.7, 5.1, 1.9]])
``` 

#### Steps/Code to Reproduce

Consider an subset of 0-1 targets of iris, where we introduce a duplicate on purpose:

```
import numpy as np
import scipy.sparse as sparse

perm = np.array([18, 8, 18, 93, 77, 81, 81])
from sklearn.datasets import load_iris
X = load_iris().data[perm]
y = load_iris().target[perm]
```

Define function to evaluate quadratic programming objective function:

```
import sklearn
import sklearn.svm

def compute_obj_func(clf, X, y):
    y_c = y.copy()
    y_c[y==0] = -1
    Kmat = sklearn.metrics.pairwise.rbf_kernel(clf.support_vectors_, gamma = clf._gamma)
    # support coefficients is alpha * y
    dc = clf.dual_coef_
    if sparse.issparse(dc):
        dc = dc.toarray()
    ys = y_c[clf.support_]
    assert np.all(dc * ys >= 0)
    assert np.all(dc * ys <= clf.C), "{}, {}".format(dc * ys, clf.C)
    assert np.allclose( np.dot(dc, np.ones_like(clf.support_)), 0)
    aQa = np.dot(np.dot(dc, Kmat), dc.T)
    eTa = np.dot(dc, ys)
    return (0.5 * aQa - eTa)
```

Now run

```
idx = np.arange(len(y))
pos_all = np.arange(len(y))
rng = np.random.RandomState(1234)

kw_args = {'gamma': 0.12682018203120274, 'tol': 1e-10, 'decision_function_shape': 'ovo'}

for i, p in enumerate([pos_all]):
    ii = idx[p].copy()
    XX = X[p].copy()
    yy = y[p].copy()

    clf_sp = sklearn.svm.SVC(**kw_args).fit(sparse.csr_matrix(XX), yy)
    clf = sklearn.svm.SVC(**kw_args).fit(XX, yy)

    if not (len(clf.support_) == len(clf_sp.support_) and
            np.allclose(clf.support_, clf_sp.support_) ):
        print("failed for index {}. Supports: {} and {}, objective functions: {}".format(
                i, ii[clf.support_], ii[clf_sp.support_], (compute_obj_func(clf, XX, yy).tolist(),
                 compute_obj_func(clf_sp, XX, yy).tolist()) ))
```

With run with `daal4py` [patches](https://github.com/IntelPython/daal4py/pull/15) the dense result is computed by Intel DAAL, while sparse result is computed by libSVM. 

DAAL find `[0 1 2 3 4 6]` as support vectors, libSVM gives `[0 1 2 3 4 5 6]`, with objective functions being 
``([[-2.249185019516638]], [[-2.2491850195166347]])`` respectively.

As you see, objective function on DAAL's solution is several machine epsilons (`14*np.finfo(np.double).eps` to be exact) lower than libSVM's objective function, and hence are essentially equivalent.

#### Expected Results

I hope to have convinced the community to either not use inputs with duplicate samples for SVC tests, or modify that test to compare values of the objective function at the solutions, or the result of 
`predict`.


#### Versions

I used scikit-learn `0.20.1`  with Numpy 1.15.4 on 64-bit Linux machine.
I also used Intel DAAL 2019.1
user: oleksandr-pavlyk
issue_12711: state: open
title: Feature Request: Tangent distance as an option in sklearn.neighbors.DistanceMetric
body: It would be nice to have 'tangent distance' as a possible metric in nearest neighbors models.

It is not a [new concept][1] but is [widely cited][2]. It is also relatively standard, the [Elements of Statistical Learning][4] covers it.

Its main use is in patter/image recognition where it tries to identify invariances of classes (e.g. the shape of '3' regardless of rotation, thickness, etc). 

![tangent distance](https://i.imgur.com/8s8rwmu.png)

I have looked for other ways to implement it in python, but [I did not find anything yet][3].

Would be happy to contribute but do not really know how to do it.

[1]: https://pdfs.semanticscholar.org/9ad2/80e4b58a9753eb7f331433efdd82efefc714.pdf
[2]: https://scholar.google.hu/scholar?q=%22tangent+distance%22
[3]: https://datascience.stackexchange.com/questions/41954/how-can-i-implement-tangent-distance-for-k-nearest-neighbor-in-python-scikit-lea
[4]: https://web.stanford.edu/~hastie/ElemStatLearn/contents.pdf
user: nocibambi
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_12697: state: open
title: BallTree assumes samples can be added component-wise
body: I am using BallTree with a custom metric and am running into the following issue: BallTree seems to assume in lines 71-92 of [this file](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/ball_tree.pyx) (possibly elsewhere also) that the sample vectors can be added component-wise. 

For my custom metric, this does not necessary produce a valid sample, i.e. the points cannot be averaged component-wise because this makes no sense for the space they live in, and thus BallTree is invalid.

Instead of averaging the samples, for cases like mine it would seem like BallTree should be restricted to using a representative central element for the dataset (a "medoid" rather than a mean). It would be great to either have this functionality implemented or for the documentation to specify that BallTree makes an explicit (in principle unnecessary) assumption that it makes sense to average samples component-wise. 
user: pkomiske
issue_12694: state: open
title: Revise plot_out_of_core_classification.py
body: In this program, Sir Eustache Diemert used first 1000 samples to measure accuracy. 
I put my efforts to extend the same program, to separate Train & Test Datasets as per guideline mentioned in README.txt file listed in Reuters-21578 datasets as provided by the UCI ML repository. Test Datasets used to measure accuracy.

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.


#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: paragmoteria
issue_12675: state: open
title: Use working_memory in mlp.predict_proba etc
body: Running predict with lots of data can blow up the ram, even on tiny models.
See https://github.com/amueller/introduction_to_ml_with_python/issues/67

It might be nice to use ``working_memory`` to restrict the size of batches used during prediction.
user: amueller
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_12662: state: open
title: LDA covariance shrinkage 'auto' bug
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
The current implementation of shrinkage 'auto' is to calculate ledoit-wolf shrinkage factor separately for each matrix. 

The within-class covariance matrices are calculated with different shrinkage factors, then summed together to be S_w. Because shrinkage factor depends largely on sample size, the shrinkage factor of every within-class covariance matrix will be much larger than that of the total covariance matrix, meaning diagonal entries are weighted more in S_w than in S_t. This will result in negative diagonal entries for the between-class covariance matrix $S_b=S_t-S_w$

negative variance then leads to weird behavior of eigen-value decomposition, including 1) wrong/meaningless explained_variance 2) questionable eigenvectors

this problem can be resolved by calculating the ledoit-wolf shrinkage factor first and pass it as an argument to the LDA decoder.

I'm not sure whether this is an intended feature or a bug.

If it's the former case, how should we interpret the eigenvectors?


user: xiaoxionglin
issue_12652: state: open
title: Warnings in common tests about set_params
body: There's a bunch of "ValueError occurred during set_params.", introduced in #7760. This doesn't look good imho because it probably means we're violating API somewhere. We should investigate.
user: amueller
labels: module:test-suite: everything related to our tests
issue_12635: state: open
title: [WIP] pairwise_distances(X) should always have 0 diagonal  intial commit
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs

Fix #12628
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
I Add a method `set_diagonal_pairwise(distances, X, Y)` that will check if the conditions are met and then set diagonal to 0.0 using: `np.fill_diagonal` (https://github.com/scikit-learn/scikit-learn/issues/12628#issuecomment-440648630)

I decided create a method to don't add logical code to the methods.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: eamanu
labels: Needs Decision: Requires decision
issue_12628: state: open
title: pairwise_distances(X) should always have 0 diagonal
body: In the case of euclidean distances, we explicitly set the diagonal of unary pairwise distances to 0: https://github.com/scikit-learn/scikit-learn/blob/e170d479acff31aadd9d2ab404952b23f7994aab/sklearn/metrics/pairwise.py#L257 as well as in pairwise_distances_chunked: https://github.com/scikit-learn/scikit-learn/blob/e170d479acff31aadd9d2ab404952b23f7994aab/sklearn/metrics/pairwise.py#L1279. We also do so for cosine distances: https://github.com/scikit-learn/scikit-learn/blob/e170d479acff31aadd9d2ab404952b23f7994aab/sklearn/metrics/pairwise.py#L550.

We should zero the diagonal of the output for all metrics through pairwise_distances and pairwise_distances_chunked (where `Y is None or Y is X`) to reduce the effect of imprecision during distance calculation.

That is, for all `metric` we eventually want:
```py
assert not np.any(pairwise_distances(X, metric=metric)[np.diag_indices(X.shape[0])])
```
and a similar assertion for `pairwise_distances_chunked`

I say eventually because I propose that:

* if current output satisfies `np.allclose(pairwise_distances(X, metric=metric)[np.diag_indices(X.shape[0])], 0)`, we just set it to 0
* if current output does not satisfy that condition, we only raise a FutureWarning saying that "The specified metric is not a valid dissimilarity metric; it does not return metric(x, x) == 0 for some values. In version 0.23, the diagonal of pairwise_distances(X) will be set to 0." We do this because at the moment we do not strictly require that the metric be a valid dissimilarity measure.
* we do the same for `pairwise_distances_chunked`
user: jnothman
issue_12614: state: open
title: Should cross validation shuffle by default?
body: Apologies if issues is the wrong place for this, but it didn't seem like a usage question...

Recently I was teaching a bunch of clinical scientists how to use scikit-learn by having them build classifiers for their datasets. Several of them ran into problems with cross validation giving terrible results with some classifiers (decision trees/random forests seem particularly bad), because their datasets were ordered, either by class or by one of the features. This led me to wonder: is there a reason why the various cross-validators (KFold, etc.) do not shuffle the data by default? It seems like this would avoid problems for novice users with real-world datasets (which are often at least partially ordered). Is there a drawback that I'm not seeing? I did a quick test with the iris dataset and couldn't see any performance cost. 
user: mojones
issue_12604: state: open
title: PLSRegression incompatibility with MultiOutputRegressor
body: #### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->

It seems that PLSRegression doesn't work with MultiOutputRegressor. This is not usually needed, but if you want to analyze what is the influence of doing multi output fitting with PLS it is needed.

#### Steps/Code to Reproduce
I got the error on my own code, but was able to reproduce it based on the Scikit-learn example availble on: https://scikit-learn.org/stable/auto_examples/ensemble/plot_random_forest_regression_multioutput.html#sphx-glr-auto-examples-ensemble-plot-random-forest-regression-multioutput-py

I just replaced all Random Forest Models with PLSRegression(n_components=1).

Here it is:

```

import matplotlib.pyplot as plt
import numpy as np
from sklearn.cross_decomposition import PLSRegression
from sklearn.model_selection import train_test_split
from sklearn.multioutput import MultiOutputRegressor

# Create a random dataset
rng = np.random.RandomState(1)
X = np.sort(200 * rng.rand(600, 1) - 100, axis=0)
y = np.array([np.pi * np.sin(X).ravel(), np.pi * np.cos(X).ravel()]).T
y += (0.5 - rng.rand(*y.shape))

X_train, X_test, y_train, y_test = train_test_split(
    X, y, train_size=400, test_size=200, random_state=4)

max_depth = 30
regr_multirf = MultiOutputRegressor(PLSRegression(n_components=1))
regr_multirf.fit(X_train, y_train)

regr_rf = PLSRegression(n_components=1)
regr_rf.fit(X_train, y_train)

# Predict on new data
y_multirf = regr_multirf.predict(X_test)
y_rf = regr_rf.predict(X_test)

# Plot the results
plt.figure()
s = 50
a = 0.4
plt.scatter(y_test[:, 0], y_test[:, 1], edgecolor='k',
            c="navy", s=s, marker="s", alpha=a, label="Data")
plt.scatter(y_multirf[:, 0], y_multirf[:, 1], edgecolor='k',
            c="cornflowerblue", s=s, alpha=a,
            label="Multi RF score=%.2f" % regr_multirf.score(X_test, y_test))
plt.scatter(y_rf[:, 0], y_rf[:, 1], edgecolor='k',
            c="c", s=s, marker="^", alpha=a,
            label="RF score=%.2f" % regr_rf.score(X_test, y_test))
plt.xlim([-6, 6])
plt.ylim([-6, 6])
plt.xlabel("target 1")
plt.ylabel("target 2")
plt.title("Comparing random forests and the multi-output meta estimator")
plt.legend()
plt.show()
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->

No error is thrown

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->

It seems that the output produced by `y_multirf = regr_multirf.predict(X_test)` has shape (1, 200, 2)  instead of (200, 2) 

```
Traceback (most recent call last):
  File "C:/Users/mijoao/.PyCharmCE2018.2/config/scratches/scratch.py", line 36, in <module>
    label="Multi RF score=%.2f" % regr_multirf.score(X_test, y_test))
  File "D:\Joao\venvs\myenv\lib\site-packages\sklearn\multioutput.py", line 286, in score
    multioutput='uniform_average')
  File "D:\Joao\venvs\myenv\lib\site-packages\sklearn\metrics\regression.py", line 534, in r2_score
    y_true, y_pred, multioutput)
  File "D:\Joao\venvs\myenv\lib\site-packages\sklearn\metrics\regression.py", line 75, in _check_reg_targets
    check_consistent_length(y_true, y_pred)
  File "D:\Joao\venvs\myenv\lib\site-packages\sklearn\utils\validation.py", line 230, in check_consistent_length
    " samples: %r" % [int(l) for l in lengths])
ValueError: Found input variables with inconsistent numbers of samples: [200, 1]


```

#### Versions
```
System
------
    python: 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 03:37:03) [MSC v.1900 64 bit (AMD64)]
executable: d:Joao\venvs\myenv\scripts\python.exe
   machine: Windows-7-6.1.7601-SP1

BLAS
----
    macros: 
  lib_dirs: 
cblas_libs: cblas

Python deps
-----------
       pip: 10.0.1
setuptools: 39.1.0
   sklearn: 0.20.0
     numpy: 1.15.2
     scipy: 1.1.0
    Cython: None
    pandas: 0.23.4

```

user: Joao-M-Almeida
issue_12600: state: open
title: sqeuclidean metric is much slower than euclidean in pairwise_distances
body: Currently, passing `metric='sqeuclidean'` to `pairwise_distances` falls back to scipy pairwise distance calculations instead of using the fast (and less accurate) implementation in scikit-learn. As a result, it can be up to 10x slower in some cases,

```py
In [1]: from eucl_dist.cpu_dist import dist                                                                                                                  
    ... import numpy as np                                                                                                                                   
In [2]: rng = np.random.RandomState(1)                                                                                                                        
    ... a = rng.rand(1000, 300)                                                                                                                              
    ...b = rng.rand(1000, 300)
In [3]: %timeit pairwise_distances(a, b, metric='euclidean')**2                                                                                             
21.5 ms ± 804 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [4]: %timeit pairwise_distances(a, b, metric='sqeuclidean')                                                                                              
212 ms ± 2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

this is particularly confusing because people who would use `sqeuclidean` will mostly do it for performance reasons,  I think,  to save one square root computation with repsect to `euclidean`.
user: rth
issue_12570: state: open
title: LinearSVC check fails on sparse succeeds on dense
body: Minor issue

```
from sklearn.datasets import make_classification
import scipy.sparse
from sklearn.svm import LinearSVC
from sklearn.utils import shuffle
import numpy as np
X, y = make_classification(n_samples=10, n_features=100, n_informative=30, n_classes=3, random_state=1)
y = np.atleast_2d(y).T
y_sparse = scipy.sparse.lil_matrix(y)
svc = LinearSVC()

# works
svc.fit(X, y)

# fails
svc.fit(X, y_sparse)
```
Yields

ValueError: Found input variables with inconsistent numbers of samples: [10, 1]

I looked at the code and there are a number of switches like "accept_large_sparse" and other sparse checks so it's not clear what was meant there or where the right place is to make the change in the shared utils/.

Basically looks like while in dense case it just throws the usual warning about y should be 1d, in sparse it throws. 

This comes up in when you use the MultiOutputClassifier for example.


user: cottrell
issue_12545: state: open
title: ColumnTransformer with array of dicts
body: Love the new columntransformer and using it.

However i'd like to be able to use it in a production environment with arrays of python dicts or similar so i dont have to install pandas in production, this does not seem possible currently, are there any plans to support it?
user: mikkelam
issue_12505: state: open
title: MLPClassifier supports fitting on multilabel output but cannot be used with partial_fit
body: #### Description
Performance is much worse when using partial_fit method on multilabel y than using fit on the same data. I suspect that the issue is partial_fit supports multi-class but not multi-label. Why is this the case when fit supports multi-label? 

#### Steps/Code to Reproduce
```python
X_train.shape, y_train.shape # --> ((3963, 4572), (3963, 39))
# where y is binary [0,1] for each of the 39 columns

mlp = MLPClassifier(hidden_layer_sizes=(500, ), activation='relu', solver='adam', alpha=0.0001, batch_size='auto', learning_rate='constant', learning_rate_init=0.001, power_t=0.5, max_iter=500, shuffle=True, random_state=123, tol=0.0001, early_stopping=False, validation_fraction=0.1, beta_1=0.9, beta_2=0.999, epsilon=1e-08, n_iter_no_change=10)
mlp.partial_fit(X_train, y_train, classes=list(range(y_train.shape[1])))
y_pred = mlp.predict(X_test)
precision_score(y_test, y_pred, average='weighted')
```

#### Expected Results
precision score is 0.635 when using just fit. 
#### Actual Results
precision score is 0.216 when using partial_fit method. 

#### Versions
System
------
    python: 3.6.5 (default, Apr 25 2018, 14:23:58)  [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)]
executable: /Users/clhq/.local/share/virtualenvs/LTR-looks-tag-recommender-QOgMS24J/bin/python
   machine: Darwin-17.7.0-x86_64-i386-64bit

BLAS
----
    macros: NO_ATLAS_INFO=3, HAVE_CBLAS=None
  lib_dirs: 
cblas_libs: cblas

Python deps
-----------
       pip: 18.1
setuptools: 40.4.3
   sklearn: 0.20.0
     numpy: 1.15.3
     scipy: 1.1.0
    Cython: None
    pandas: 0.23.4

user: christinebuckler
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_12491: state: open
title: check_estimator not valid for vectorizers
body: [`check_estimator`](https://github.com/scikit-learn/scikit-learn/blob/f6b0c67290d3cbc8b099eb7559b4f42b84af2584/sklearn/utils/estimator_checks.py#L273) is a really powerful tool for deciding scikit-compatibility of estimators through automated checking.

Although this is true, it restricts estimators that do not expect as input `numpy arrays of shape [n_samples, n_features]`, but vector of objects (i.e. Vectorizers) to use this tool.

Mainly, being redirected from my issue in [sklearn-template](https://github.com/scikit-learn-contrib/project-template/issues/29), it would be really helpful if all the constant input needed inside `check_estimator` could be collected as the values of an object or instantiated as the return values of a function that the user of `check_estimator` would be able to define, following a certain specification.

An argument can rise for that, if someone considers that any [Transformer](http://scikit-learn.org/stable/modules/generated/sklearn.base.TransformerMixin.html#sklearn.base.TransformerMixin.fit_transform) requires as input `numpy arrays of shape [n_samples, n_features]`. But if we approach a very common `Transformer` in sklearn, namely [Tf-idf](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html) we can see that its input is not an `np.array`.

To sum up, as scikit-learn aims to be a more and more general library that can standardize a template for developing machine learning packages in python, it would be really essential to support various input formats on the automated checking of consistency of its basic object, the Estimator.

Thank you for your attention!
user: ysig
issue_12484: state: open
title: RFE: more average_method's for normalized_mutual_info_score
body: #### Description

It would be nice if [`normalized_mutual_info_score `](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.normalized_mutual_info_score.html) could compute [uncertainty coefficient, AKA proficiency](https://en.wikipedia.org/wiki/Uncertainty_coefficient).

#### Implementation

E.g., `average_method` can be `labels_true` to compute proficiency.

#### Expected Results

```
normalized_mutual_info_score(
    labels_true=[0,1,0,1], 
    labels_pred=[0,1,0,2], 
    average_method='labels_true')
```
should evaluate to 1.

user: sam-s
issue_12480: state: open
title: haversine distance for OPTICS
body: When i am trying to run `OPTICS(min_samples=2, max_eps=epsilon, metric='precomputed', algorithm='ball_tree', rejection_ratio=0.1 , n_jobs=3).fit(distance_matrix)` i am getting **Metric 'precomputed' not valid. Use sorted(sklearn.neighbors.VALID_METRICS['ball_tree']) to get valid options. Metric can also be a callable function.** where `distance_matrix` is a numpy array of harvesine distances.

Again when trying `OPTICS(min_samples=2, max_eps=epsilon, metric='haversine', algorithm='ball_tree', rejection_ratio=0.1 , n_jobs=3).fit(np.radians(coordinates[:, [0, 1]]))` i am getting  **ValueError: Unknown metric haversine.**

Using _Version: 0.21.dev0_
user: koushiksaha89
issue_12476: state: open
title: [MRG] Add early exaggeration iterations as argument to t-SNE
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.

Converting early exaggeration iterations from a private constant to a public member variable of the TSNE class. The default value for this new argument is set at 250 such that code using previous versions of sklearn will still get identical results if no action is taken to account for the change.

Being able to set this variable is an important component of running t-SNE and it shouldn't be hidden. Other implementations ([e.g., LvdM's](https://github.com/lvdmaaten/bhtsne/blob/master/tsne.h#L44-L45)) expose it. In the LvdM case it's actually via two arguments but sklearn doesn't distinguish between momentum switch and stop lying. It doesn't seem necessary to treat the two arguments differently.

This change is motivated by [recent work](https://doi.org/10.1101/451690) that shows a high quality embedding can be achieved with much fewer than 250 early exaggeration iterations.

#### Any other comments?

Validated that results are identical before and after change for the same inputs.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: cciccole
issue_12468: state: open
title: Univariate feature selection methods should allow passing kwargs to score_func callable
body: Univariate feature selection methods, SelectKBest/Percentile/Fdr/Fpr/Fwe and GenericUnivariateSelect does not have functionality to allow passing of score_func kwargs, some of which are important.

One thought too is that an implementation should be GridSearchCV compliant such that score_func kwargs parameters can be searched.
user: hermidalc
issue_12464: state: open
title: Issues with negative values in sample_weight
body: #### Description
I am not sure what the interpretation of a negative value in `sample_weight` might be and why this should be supported, but I believe that there should be constraints in using non-negative values in several cases; the use of negative ones can lead to some very strange results. 

See an example below for `r2_score` where the use of negative weights yields a value larger than one, which really does not make sense. 

#### Steps/Code to Reproduce
```python
import numpy as np

from sklearn.metrics import r2_score

np.random.seed(seed=2)
x = np.random.randn(100,)
y = x + 0.3*np.random.randn(*x.shape)
w = np.random.randn(*x.shape)

r2_score(x, y, sample_weight=w)
```

#### Expected Results
Something smaller or equal to 1.0

#### Actual Results
1.1919195778883198

#### Versions
System:
    python: 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]
executable: C:\Users\nak142\Miniconda3\envs\sklearn_contrib\pythonw.exe
   machine: Windows-10-10.0.17134-SP0

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: C:/Users/nak142/Miniconda3/envs/sklearn_contrib\Library\lib
cblas_libs: mkl_rt

Python deps:
       pip: 10.0.1
setuptools: 40.0.0
   sklearn: 0.21.dev0
     numpy: 1.15.0
     scipy: 1.1.0
    Cython: 0.28.5
    pandas: None

<!-- Thanks for contributing! -->

user: agamemnonc
issue_12457: state: open
title: [MRG] Conditional GMM sampling
body: #### What does this implement?
GMMs can be used as conditional generative models by conditioning on a subset of the dimensions. This feature is missing at the moment i scikit-learn, which only supports sampling from the full (un-conditioned) distribution.

The added functionality allows to select a subset of the indices of the modeled variable, along with values of these dimensions, and sample from the remaining dimensions.
user: isacarnekvist
issue_12444: state: open
title: GPR not optimizing
body: #### Description
When using GPR, it currently does NOT optimize the hyperparameters when kernel is set to "None" as it says it is should. (GaussianProcessRegressor from gpr.py)

#### Steps/Code to Reproduce
The code below should produce some GPR interpolation, but if you change the y values you can notice that the kernel doesn't change. This is because the optimization step is not being performed in grp.py.
```py
import numpy as np
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, WhiteKernel
#from __future__ import division

x = np.random.rand(10,2)
y = np.array(np.exp(range(0,10)))
y2 = np.array(np.exp(range(-10,0)))

GP = GaussianProcessRegressor(n_restarts_optimizer=20)
GP.fit(x,y)
print GP.kernel_
GP.fit(x,y2)
print GP.kernel_
```

What I did to fix this: 
The commented code is what WAS there (line 173-175 of gpr.py) and the uncommented code is what I replaced it with. declaring the bounds as "fixed" overrides the ability to optimize. 
```py
        # if self.kernel is None:  # Use an RBF kernel as default
        #     self.kernel_ = C(1.0, constant_value_bounds="fixed") \
        #         * RBF(1.0,)
        if self.kernel is None:  # Use an RBF kernel as default
            self.kernel_ = C(1.0) \
                * RBF(1.0)
```


user: klinnell
issue_12442: state: open
title: OPTICS Benchmark and improve the performance
body: Our OPTICS is now O(n^2). We can improve it to O(n * logn) with a heap (See ELKI and R dbscan).
Open this issue for us to benchmark our implementation and (maybe) try to improve it in the future.

Related comments:
* (from kno10) It is worth noting that because of this operation, the current OPTICS in scikit-learn obviously needs n² operations, while using a heap here would reduce this to n*O(log n) in this particular location. At the same time, the n² part has very low constant factors (because of numpy), while attempting to implement an *updateable* heap efficiently can easily introduce much larger constant factors here...
* (from kno10) https://github.com/scikit-learn/scikit-learn/pull/12421#issuecomment-431708045

ping @espg you can post your results here.
user: qinhanmin2014
issue_12437: state: open
title: Estimators with metric='euclidean' default but supporting metric_params and p should instead have metric='minkowski'
body: Where estimators can be configured for metric, the point of providing a parameter `p` is that it should be possible to switch between Manhattan and Euclidian and other Minkowski distances just by modifying `p` (without changing `metric`). However this is not possible where `metric='euclidean'` by default. Rather, where `p` is available, `metric` should be 'minkowski' by default.

We should change the default metric in OPTICS without deprecation as it has not been released. Elsewhere, we need to:
* Warn for two versions if the user has not set metric but has set `p != 2`
* Change the default `metric` to minkowski when that deprecation period is over.
user: jnothman
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_12433: state: open
title: Add built-in dataset with missing values and categorical data?
body: Follow up on #12424:
We don't have a built-in dataset with missing values and/or mixed feature types. I think it would be good to have one to have non-synthetic examples for the column transformer and imputation strategies.
While I like to reduce the number of custom fetchers and would like to use ``fetch_openml`` as much as possible, I think there's a benefit to having a built-in dataset so that examples can run without internet connection.

Not sure what good candidates would be. Titanic is somewhat obvious though I'm not sure about missingness patterns there. Adult would be nice but might be too large to ship (4mb - would double the size of the wheel so seems unreasonable).

Maybe the ames housing data would be appropriate, not sure if it has missing values.
user: amueller
issue_12425: state: open
title: Feature Request: Optimal Cut-offs for F1 score, accuracy for multi-class
body: Dear All,

For multi-class classifiers, we do not know what probability value will maximize the accuracy or F1 score. Or any metric for which you need to trade-off between FP and FN. So we wrote a small script that uses a brute force optimizer to find that value for the metrics. 

https://github.com/soodoku/optimal_classification_cutoffs

It may be a useful thing to add to the core functions.

<!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->


<!-- Thanks for contributing! -->

user: soodoku
issue_12408: state: open
title: Sklearn.svm.SVC,the "predict" method predict all samples to  same one label, which is wrong. 
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
I want to use sklearn.svm.SVC to predict probality of each label. However, when I use the method of "predict", the SVC estimator will predict all samples to the same label whether I set the probablity to True or False. 
When I replace SVC with LinearSVC, the result becomes normal.
the following "Expected Results" was run by replacing the SVC(probablity=True) with LinearSVC().

#### Steps/Code to Reproduce

```python
from sklearn.metrics import confusion_matrix
from sklearn import datasets, linear_model
from sklearn.svm import SVC, LinearSVC, NuSVC

# some code about to load X and y

clf = SVC(probability=True)
clf.fit(X, y)
y_pred = clf.predict(X)
print(clf.classes_)
print('the true label:',y[:20])
print('the predict label:', clf.predict(X)[:20])
```

#### Expected Results
['pop_and_registration' 'recall' 'repair_process' 'warranty_extension'
 'warranty_lookup' 'warranty_policy']
the true label: ['warranty_lookup' 'warranty_lookup' 'warranty_lookup' 'warranty_lookup'
 'warranty_policy' 'warranty_policy' 'warranty_policy' 'warranty_policy'
 'warranty_policy' 'warranty_policy' 'pop_and_registration'
 'pop_and_registration' 'warranty_policy' 'pop_and_registration'
 'warranty_extension' 'warranty_extension' 'warranty_extension'
 'warranty_policy' 'warranty_lookup' 'warranty_lookup']
the predict label: ['warranty_lookup' 'warranty_lookup' 'warranty_lookup' 'warranty_lookup'
 'warranty_policy' 'warranty_policy' 'warranty_policy' 'warranty_policy'
 'warranty_policy' 'warranty_policy' 'pop_and_registration'
 'pop_and_registration' 'warranty_policy' 'pop_and_registration'
 'warranty_extension' 'warranty_extension' 'warranty_extension'
 'warranty_policy' 'warranty_lookup' 'warranty_lookup']


#### Actual Results
['pop_and_registration' 'recall' 'repair_process' 'warranty_extension'
 'warranty_lookup' 'warranty_policy']
the true label: ['warranty_lookup' 'warranty_lookup' 'warranty_lookup' 'warranty_lookup'
 'warranty_policy' 'warranty_policy' 'warranty_policy' 'warranty_policy'
 'warranty_policy' 'warranty_policy' 'pop_and_registration'
 'pop_and_registration' 'warranty_policy' 'pop_and_registration'
 'warranty_extension' 'warranty_extension' 'warranty_extension'
 'warranty_policy' 'warranty_lookup' 'warranty_lookup']
the predict label: ['pop_and_registration' 'pop_and_registration' 'pop_and_registration'
 'pop_and_registration' 'pop_and_registration' 'pop_and_registration'
 'pop_and_registration' 'pop_and_registration' 'pop_and_registration'
 'pop_and_registration' 'pop_and_registration' 'pop_and_registration'
 'pop_and_registration' 'pop_and_registration' 'pop_and_registration'
 'pop_and_registration' 'pop_and_registration' 'pop_and_registration'
 'pop_and_registration' 'pop_and_registration']



#### Versions
System
------
   machine: Windows-10-10.0.17134-SP0
    python: 3.5.6 |Anaconda, Inc.| (default, Aug 26 2018, 16:05:27) [MSC v.1900 64 bit (AMD64)]
executable: D:\ProgramData\Anaconda3\envs\py35\python.exe

BLAS
----
cblas_libs: cblas
    macros: 
  lib_dirs: 

Python deps

-----------

     numpy: 1.15.2
     scipy: 1.1.0
   sklearn: 0.20.0
setuptools: 40.2.0
    Cython: None
    pandas: 0.23.4
       pip: 10.0.1


user: yucc2018
issue_12392: state: open
title: Target column names in datasets provided by Sklearn
body: As far as I know, Sklearn does not provide a way to get the names of the "columns" corresponding to the target. It would be useful to choose an standard name for this data, so that Sklearn-compatible datasets can provide this information.

This has been asked before in StackOverflow:
https://stackoverflow.com/questions/50877785/how-to-obtain-target-feature-name-from-a-sklearn-bunch
but I have not seen any issue here. However in PR https://github.com/scikit-learn/scikit-learn/pull/2865 it was hinted that it could be useful. Also, I think that if in the future Sklearn provides multi-output datasets it will be necessary.

I propose that this data will be made available as "target_feature_names", as it seems clear what that means.


user: vnmabus
issue_12385: state: open
title: Function to get scorers for task
body: 
I would like to see a utility which would construct a set of applicable scorers for a particular task, returning a Mapping from string to callable scorer. It will be hard to design the API of this right the first time. [Maybe this should be initially developed outside this project and contributed to scikit-learn-contrib, but I think it reduces risk of mis-specifying scorers, so it's of benefit to this project.]

The user will be able to select a subset of the scorers, either with a dict comprehension or with some specialised methods or function parameters. Initially it wouldn't be efficient to run all these scorers, but hopefully we can do something to fix #10802 :|.

Let's take for instance a binary classification task. The function `get_applicable_scorers(y, pos_label='yes')` for binary `y` might produce something like:
```
{
    'accuracy': make_scorer(accuracy_score),
    'balanced_accuracy': make_scorer(balanced_accuracy_score),
    'matthews_corrcoef': make_scorer(matthews_corrcoef),
    'cohens_kappa': make_scorer(cohens_kappa),
    'precision': make_scorer(precision_score, pos_label='yes'),
    'recall': make_scorer(recall_score, pos_label='yes'),
    'f1': make_scorer(f1_score, pos_label='yes'),
    'f0.5': make_scorer(f1_score, pos_label='yes', beta=0.5),
    'f2': make_scorer(f1_score, pos_label='yes', beta=2),
    'specificity': ...,
    'miss_rate': ...,
    ...
    'roc_auc': make_scorer(roc_auc_score, needs_threshold=True),
    'average_precision': make_scorer(average_precision_score, needs_threshold=True),
    'neg_log_loss': make_scorer(average_precision_score, needs_proba=True, greater_is_better=False),
    'neg_brier_score_loss': make_scorer(average_precision_score, needs_proba=True, greater_is_better=False),
}
```

Doing the same for multiclass classification would pass `labels` as appropriate, and would optionally would get per-class binary metrics, as well as overall multiclass metrics.

I'm not sure how `sample_weight` fits in here, but ha! we still don't support weighted scoring in cross validation (#1574), so let's not worry about that.
user: jnothman
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_12364: state: open
title: [WIP] priority of features in decision to splitting
body: 
#### Reference Issues/PRs
Fixes #12259 . See also  issues #2386 , #8443 and  #12188 

#### What does this implement/fix? Explain your changes.

Basically, Scikit-learn implements decision trees with 2 different types of splitters. A RandomSplitter and a BestSplitter. I focus on BestSplitter, which currently has two flaws:

1. It is random (provide different outputs on the default config). This happens when there are two or more 'best' features with equal criterion value.
2. It doesn't allow to give priors on the features, for example, when there is a Tie. It would be a nice-to-have if we could chose a features that we prefer. Either because it is easier to compute, or because it is less noisy or because it is easy to interpret. The ideal would be to encode it in the index of the column of the feature. Lower index means higher priority. 

We have a supplementary constraint which is the Shuffle constrain. Because currently there is a max_features parameter which allows us to do not test all the features at each node if there are many. So this is a nice feature and I believe we should always shuffle. The problem is that shuffling means non deterministic...


#### So we have:

- Sorted / Unsorted       -> stands for the fact of giving priority to lower index features if and only if there is a Tie.
- Deterministic / Non     -> stands for the fact of always obtaining the same result, at least for the default configuration.
- Shuffled / Looped / No  -> stands for the fact of shuffling the features when we try them. We can shuffle them randomly, we can loop through them or can test them in order (from 0 to N). This is important only if max_featues < nb_features  or if we do not sort the features.



So for me the ideal is to have a a prior on the features (Sorted, saying lower index means I prefer the feature). 
We need to do Shuffle or at least Loop, in order to support  max_features, which is often useful, and necessary for sparse cases.

Then it should be deterministic in the default configuration, so if we want this we have the following possibilities:
 - we can do looping instead of shuffling: this would make random_state useless here... so is not viable
 - we can do shuffle and sort the features to the best of our knowledge: So we do shuffling and if 2 features from the tested features are 'best' we keep the lower index. This will make the default config where `max_features=n_features` stable regardless of the `random_state` because we will test all the features.
 

So I implemented this BestSplitter2 that shuffles, and is deterministic when `max_features=n_features` and gives priority to the lower index features when there is a tie.


Here you can see an example of the previous and the new one:
Previous BestSplitter is on the left and New BestSplitter2 is on the right 

[![Screenshot-from-2018-10-12-11-31-55.png](https://i.postimg.cc/m2wbstBY/Screenshot-from-2018-10-12-11-31-55.png)](https://postimg.cc/ppmNQV9r)

We observe that BestSplitter2 give priority to lower index features.
And when we test stability 


```
import numpy as np
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier

iris = load_iris()
x,y  = iris.data, iris.target

dtc1 = DecisionTreeClassifier(random_state=1,splitter='best2')
dtc2 = DecisionTreeClassifier(random_state=2,splitter='best2')

rs = np.random.RandomState(1234)
itr = rs.rand(x.shape[0]) < 0.75

dtc1.fit(x[itr],y[itr])
dtc2.fit(x[itr],y[itr])

print(  (dtc1.predict(x[~itr]) != dtc2.predict(x[~itr])).sum() )
```

We obtain `0` as expected in #12259 


#### Any other comments?

I don't know what do you think, I think having many splitters is not a solution either.
Personally I would replace the current bestSplitter with the one described above, as they are essentially the same but with some improvements.

Thank you very much

user: GMarzinotto
issue_12358: state: open
title: [RFC] Support sparse input in OneHotEncoder
body: Currently the `OneHotEncoder` is not supporting sparse input. I see that it was mentioned in #1285 and actually I don't see why we cannot handle this case. Moreover, it seems a possible use case, isn't it?
user: glemaitre
issue_12335: state: open
title: Can't instantiate pipeline with placeholder to search in GridSearchCV
body: A cosmetic complaint:
It's not possible to instantiate a pipeline with a placeholder like in #12272 and grid-search the final estimator. If the final estimator is ``None``, there is no ``score`` method and so ``GridSearchCV`` refuses to run.
I'm not sure if this is worth fixing or how to most elegantly fix it, but it's a bit ugly.
user: amueller
issue_12326: state: open
title: [WIP] Extending AdaBoost's base estimator's feature_importances_ attr to support coef_ attr of an estimator
body: Modified `feature_importances_` attribute in AdaBoost's base estimator to work with `coef_` attributes of given estimators if they don't provide `feature_importances_`  
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
Fixes #12137 
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
`feature_importances_` attribute of `BaseWeightBoosting` class now converts `coef_` attribute of input classifiers to their respective `feature_importances_`  using existing `_get_feature_importances` method in `feature_selection/from_model.py`.

This method accepts an optional argument: `norm_order` which defaults to `2` (`L2 norm`).
```
norm_order : {non-zero int, inf, -inf, "fro"}, optional
        Same as `numpy.linalg.norm`'s norm_order parameter
```   

#### Any other comments?
This method is now renamed to `get_feature_importances` and moved to `utils/__init__.py` as it is a shared resource.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: dsprahul
issue_12325: state: open
title: Feature request: auto-saving of best solution found so far by GridSearchCv or RandomizedSearchCv
body: Sorry for posting it here as it's not a really issue, rather a missing feature. However, many times I have seen grid search jobs running for days and in vain due to power outage, memory problems etc.
As simple as it reads, could someone add saving of current model found so far, if parameter 'best_model_path' is set? This will make scikit-learn much more user friendly out of the box.
Thank you!
user: fingoldo
issue_12322: state: open
title: Joblib makes callstacks impossible to decipher
body: So I'm trying to work with a script of mine which is a pipeline containing a ColumnTransformer in cross-validation. I'm trying to do some profiling, but each of these three things calls into joblib, which adds 3-4 layers *each* to the stack. Similarly if I want to do interactive debugging, this is pretty hard to manage.

This is for n_jobs=None. I'm not sure if ``n_jobs=1`` would make it better?

I suggest we either hard-code a separate path for ``n_jobs=1`` into scikit-learn, or there could be a shortcut in joblib with a single level on the callstack.
Either way, right now this is very difficult to work with.
user: amueller
issue_12315: state: open
title: Dask Joblib backend cancels required futures
body: Here is a failing example that uses the Dask joblib backend to parallelize a nested RandomForestClassifier within a RandomSearchCV

```python
from scipy.stats import randint as sp_randint

from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
from sklearn.datasets import load_digits
from sklearn.ensemble import RandomForestClassifier

from dask.distributed import Client

def test_random_forest_random_search():
    with Client(processes=False) as client:
        # Taken from http://scikit-learn.org/stable/auto_examples/model_selection/plot_randomized_search.html

        # get some data
        digits = load_digits()
        X, y = digits.data, digits.target

        # build a classifier
        clf = RandomForestClassifier(n_estimators=20)
        # specify parameters and distributions to sample from
        param_dist = {"max_depth": [3, None],
                      "max_features": sp_randint(1, 11),
                      "min_samples_split": sp_randint(2, 11),
                      "bootstrap": [True, False],
                      "criterion": ["gini", "entropy"]}

        n_iter_search = 20
        random_search = RandomizedSearchCV(clf, param_distributions=param_dist,
                                           n_iter=n_iter_search, cv=5)
        from sklearn.externals import joblib
        with joblib.parallel_backend('dask'):
            random_search.fit(X, y)
```

### What happened

By placing a breakpoint here:

```diff
diff --git a/distributed/client.py b/distributed/client.py
index f9cf7f94..e47d0d16 100644
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -314,6 +314,8 @@ class Future(WrappedKey):
     def release(self, _in_destructor=False):
         # NOTE: this method can be called from different threads
         # (see e.g. Client.get() or Future.__del__())
+        if not default_client().cluster.scheduler.story(self.key):
+            import pdb; pdb.set_trace()
         if not self._cleared and self.client.generation == self._generation:
             self._cleared = True
             try:
```

We find that Joblib has stopped the computation

https://github.com/scikit-learn/scikit-learn/blob/3804ccd2770ac4c026a823d019091917e4a2c70e/sklearn/externals/joblib/parallel.py#L1002-L1004

Which then goes ahead and clears out live futures

https://github.com/scikit-learn/scikit-learn/blob/3804ccd2770ac4c026a823d019091917e4a2c70e/sklearn/externals/joblib/_dask.py#L156-L159

This then fails in retrieve when we get the job result

https://github.com/scikit-learn/scikit-learn/blob/3804ccd2770ac4c026a823d019091917e4a2c70e/sklearn/externals/joblib/parallel.py#L901

I'm curious why we're stopping things while we still have live futures.  Have we found a "good enough" result and are stopping early?  If so, should we be getting the results of the submitted-but-discarded futures?  Is there something that Dask can do to help here?
user: mrocklin
issue_12308: state: open
title: test_sample_order_invariance in common metrics tests applied to threshold metrics
body: This sample invariance test is currently applied to the ranking tests, and that seems strange.
For ranking the scores should have a different shape and be integer (right now y_true and y_pred are binary so it's valid scores but it's not a great test).

Also, the same should be tested for multi-class.
Via #10481.
user: amueller
labels: module:test-suite: everything related to our tests
issue_12297: state: open
title: `prefit` option missing for voting?
body: I work on python 3.6.5 and sklearn 0.19.1 on conda 4.5.11.

According to this [post](https://stackoverflow.com/questions/42920148/using-sklearn-voting-ensemble-with-partial-fit), I am surprised that voting classifiers do not accept `prefix` classifiers. 

May I know is it because it is unnecessary from your respective or it was purely short of hand?
Thank you very much.
user: tengerye
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_12259: state: open
title: DecisionTreeClassifier behaviour when there are 2 or more best splitter (a tie among splitters) 
body: #### Description

My issue has been discussed in the past issues #2386  and #8443

It concerns `DecisionTreeClassifier` and the `splitter` parameter
In the [docs](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.decision_path) this parameter has two possible values  `splitter="best"` and `splitter="random"`


The current behaviour when  `splitter="best"` is to shuffle the features at each step and take the best feature to split. In case there is a tie, we take a random one.

In some cases, there is a prior on the feature importance or in the ease of the interpretation. 

For example, some features can be noisy (due to possible data errors or any other cause) and shouldn't be used for splitting unless there is no 'good quality' feature available that does the job. Similarly, some features can be easy to understand and some others can be obscure and shouldn't be used for splitting when possible.


The current implementation doesn't allow having this kind of prior. So when two features tie as the best splitter one get chosen at random. I believe this could be an important improvement. Specially when the trees do not have a `max_depth`, because they tend to over-fit on random features while there may be some "better" features (in terms of prior) that do the work.

Having an option such as `splitter="best_no_shuffle"` would allow the user to provide the features in order of importance and then when two or more features tie as the best splitters, it would systemically chose the features with the lowest index. This was once proposed [here](https://github.com/scikit-learn/scikit-learn/issues/2386#issuecomment-23148454) 


If you believe that this can be a valuable improvement I can try to implement it.
Or maybe you know a better workaround that solves my problem ! 






#### Steps/Code to Reproduce

I have recovered the code from  #8443 because it is simple and explains the issue


```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier

iris = load_iris()
x,y  = iris.data, iris.target

dtc1 = DecisionTreeClassifier(random_state=1)
dtc2 = DecisionTreeClassifier(random_state=2)

rs = np.random.RandomState(1234)
itr = rs.rand(x.shape[0]) < 0.75

dtc1.fit(x[itr],y[itr])
dtc2.fit(x[itr],y[itr])

print(  (dtc1.predict(x[~itr]) != dtc2.predict(x[~itr])).sum() )
```


#### Expected Results
`Should print 0`

#### Actual Results
`Prints 1`

#### Versions

System
------
   machine: Linux-4.4.0-135-generic-x86_64-with-Ubuntu-16.04-xenial
executable: /usr/bin/python3
    python: 3.5.2 (default, Nov 23 2017, 16:37:01)  [GCC 5.4.0 20160609]

BLAS
----
cblas_libs: openblas, openblas
    macros: HAVE_CBLAS=None
  lib_dirs: /usr/lib

Python deps
-----------
     scipy: 0.19.0
   sklearn: 0.21.dev0
    pandas: 0.20.3
     numpy: 1.13.3
    Cython: 0.26
setuptools: 36.7.2
       pip: 9.0.1




user: GMarzinotto
issue_12249: state: open
title: Default values for nu and tol in OneClassSVM
body: I think the default value for `nu` in the `OneClassSVM` should be 0.1 and not 0.5. As `nu` roughly corresponds to the fraction of outliers, it makes more sense to set it to a lower value.
user: albertcthomas
issue_12235: state: open
title: add clustering examples from 2017 Streaming Partition MIT/AWS/IEEE Challenge Datasets with Known Truth Partitions
body: #### Description
 There are some good test examples in 2017 Streaming Partition  MIT/AWS/IEEE Challenge Datasets with Known Truth Partitions at http://graphchallenge.mit.edu/data-sets used in  https://arxiv.org/abs/1708.07481 with  'clusterQR' to get 100% correct partitions. If using  wget.download is OK in scikit-learn code base examples, some of those would be nice and not too hard to add scikit-learn. 

#### Steps/Code to Reproduce
N/A

#### Expected Results
More large clustering tests with Known Truth Partitions

#### Actual Results
Not enough challenging clustering tests with Known Truth Partitions to evaluate clustering methods for "large" datasets with "many" clusters. 

#### Versions
N/A
user: lobpcg
issue_12228: state: open
title: Add an optional l2 regularization to the graphical Lasso
body: The graphical lasso is quite unstable on very ill-conditioned matrix. It is a well known issue that arises from the fact that it is a very optimization problem.

Adding a small amount of l2-regularization helps a lot. On a problem in my lab, it ended being the only way to estimate a sparse inverse covariance. It is actually a simple modification. If people agree, I can try to find the time to contribute this.

What do people think?
user: GaelVaroquaux
issue_12192: state: open
title: ENH: Add support for cosine distance in k-means
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

See also #1188.

#### What does this implement/fix? Explain your changes.

Adds support for cosine distance in k-means including the minibatch implementation. We trust that users know what they're doing and allow the metric to be a callable with metric_kwargs. 

#### Any other comments?

I enabled the cleaning of headers, footers, and quotes from the 20newsgroups dataset documents for the example program plot_document_clustering.py. Now it clearly shows improvements in the metrics V-measure and ARI when using cosine distance for text document clustering. 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: flaviomartins
labels: Needs Decision: Requires decision
issue_12189: state: open
title: VotingClassifier estimators_ are based on transformed labels
body: #### Description
The `VotingClassifier` transforms the labels before training the sub-estimators, so if you try to use them directly for predictions or scoring, you get unexpected results. IMHO, this should either be fixed (but I'm guessing it's a performance optimization) or at least the documentation should warn about this fact.

#### Steps/Code to Reproduce
```python
from __future__ import print_function
import numpy as np
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

X_train = np.array([[1., 2.], [3., 4.], [4., 3.], [2., 1.]])
y_train = np.array(['a', 'b', 'b', 'a'])

log_reg = LogisticRegression(multi_class="ovr", solver="liblinear")
log_reg.fit(X_train, y_train)
print("LogisticRegression score:", log_reg.score(X_train, y_train))
print("LogisticRegression predict:", log_reg.predict(X_train))

svc = SVC(gamma="auto")
svc.fit(X_train, y_train)
print("SVC score:", svc.score(X_train, y_train))
print("SVC predict:", svc.predict(X_train))

voting_clf = VotingClassifier([
    ("log_reg", LogisticRegression(multi_class="ovr", solver="liblinear")),
    ("svc", SVC(gamma="auto")),
])
voting_clf.fit(X_train, y_train)
print("VotingClassifier score:", voting_clf.score(X_train, y_train))
print("VotingClassifier predict:", voting_clf.predict(X_train))

#So far so good, now this is unexpected:
for index, estimator in enumerate(voting_clf.estimators_):
    print("Estimator", index, "score:", estimator.score(X_train, y_train))
    print("Estimator", index, "predict:", estimator.predict(X_train))
```

#### Expected Results
I would expect the sub-estimators to produce the same score and predictions as an equivalent classifier trained outside of the `VotingClassifier`.

```
LogisticRegression score: 0.5
LogisticRegression predict: ['b' 'b' 'b' 'b']
SVC score: 1.0
SVC predict: ['a' 'b' 'b' 'a']
VotingClassifier score: 1.0
VotingClassifier predict: ['a' 'b' 'b' 'a']
Estimator 0 score: 0.5
Estimator 0 predict: ['b' 'b' 'b' 'b']
Estimator 1 score: 1.0
Estimator 1 predict: ['a' 'b' 'b' 'a']
```

#### Actual Results
```
LogisticRegression score: 0.5
LogisticRegression predict: ['b' 'b' 'b' 'b']
SVC score: 1.0
SVC predict: ['a' 'b' 'b' 'a']
VotingClassifier score: 1.0
VotingClassifier predict: ['a' 'b' 'b' 'a']
Estimator 0 score: 0.0
Estimator 0 predict: [1 1 1 1]
Estimator 1 score: 0.0
Estimator 1 predict: [0 1 1 0]
/Users/ageron/.virtualenvs/ml/lib/python3.6/site-packages/sklearn/metrics/classification.py:182: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  score = y_true == y_pred
/Users/ageron/.virtualenvs/ml/lib/python3.6/site-packages/sklearn/metrics/classification.py:182: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  score = y_true == y_pred
```

#### Versions
```
Darwin-17.7.0-x86_64-i386-64bit
Python 3.6.6 (default, Jun 28 2018, 05:43:53)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]
NumPy 1.15.2
SciPy 1.1.0
Scikit-Learn 0.20.0
```

user: ageron
issue_12188: state: open
title: unstable result with Decision Tree Classifier
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
I have developed a java code to produce a decision tree without any pruning strategy. The used desicion rule is also the default majority rule. Then I opted to use python for its simplicity. The problem is the randomness in the DecisionTreeClassifier. Although splitter is set to "best" and max_features="None", so as all features are used, and random_state to 1, I don't finish with the same result as the java code generates. Exactly the same training and test data sets are used for python and java. How can I eliminate all randomness to obtain the same result with the java code please?
Help please.
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->

#### Steps/Code to Reproduce

<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
Same decision tree produced with java
#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
different confusion matrix in each time I fixed random_state to 1.
#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->





Windows-8.1-6.3.9600-SP0
Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 10:22:32) [MSC v.1900 64 bit (AMD64)]
NumPy 1.14.0
SciPy 1.0.0
Scikit-Learn 0.20.dev0


<!-- Thanks for contributing! -->

user: Ichaab
issue_12175: state: open
title: Graph lasso precision test failure on  win-32/mkl/python3.7 
body: In     ``def test_graph_lasso(random_state=0):``

https://github.com/ContinuumIO/anaconda-issues/issues/10112#issuecomment-424825573
user: amueller
issue_12162: state: open
title: Ensure ColumnTransformer also works with empty slices as column selector
body: Follow-up on https://github.com/scikit-learn/scikit-learn/pull/12084

That PR fixed the "empty column specification" problem for lists and boolean arrays, but not for slices. 
I commented there:

> @jnothman after updating this PR to also check for all-False boolean arrays, I realized that also slices can be empty .. But to fully properly determine of they are empty, you need the X data, which I actually just removed from `_iter` in the previous PR (#12107).
> Not difficult to add this back, but it also raises the question if this should only be done during fit and not anymore rely on the data during transform (similarly as we decided to not evaluate the function any more during transform). This is certainly all possible, but will complicate the code a bit further .. Thoughts ?

So we should further fix this fully check for empty column specifiers, including slices, and possibly only check for this during fit (and store the information somehow)

user: jorisvandenbossche
issue_12148: state: open
title: Improvements to check_array to handle heterogenous / object data
body: With the recent improvements to `OneHotEncoder` and `ColumnTransformer`, we now have estimators in scikit-learn that explicitly accept object-dtyped data or heterogenous data / DataFrames. In several cases, we have noted that we need to update utility functions like `check_array` to better handle those new use cases.

Some cases encountered:

* In the `OneHotEncoder`, we have some special code to deal with lists of strings / mixed numeric and string:

  https://github.com/scikit-learn/scikit-learn/blob/2d6603448a9d98212caaec9595b9b39067df54be/sklearn/preprocessing/_encoders.py#L49-L53

  This part is specifically targeting lists, where numpy `np.array([mixed list])` converts everything to string once there is at least a single string in the lists.

* Also in the `OneHotEncoder`, there is a second part to check for NaN values in case of object data:

  https://github.com/scikit-learn/scikit-learn/blob/2d6603448a9d98212caaec9595b9b39067df54be/sklearn/preprocessing/_encoders.py#L55-L58

  which could also be moved to `check_array` (see https://github.com/scikit-learn/scikit-learn/pull/12033#issuecomment-420911048)

* In the `ColumnTransformer`, we want to convert lists to arrays, but preserve DataFrames / sparse matrices (active PR: https://github.com/scikit-learn/scikit-learn/pull/12104), for which we currently are adding this workaround:

  https://github.com/scikit-learn/scikit-learn/blob/864c2cce14c87831692fb08fb782f2fd3c97b654/sklearn/compose/_column_transformer.py#L515-L519

* In the `OneHotEncoder`, we do encoding column by column, so it would not be needed to first convert a DataFrame to an array before accessing columns, we could do the checking of the data only after accessing a single column (to avoid dtype conversion). (opened a separate issue about this: https://github.com/scikit-learn/scikit-learn/issues/12147)

* In the `FunctionTransformer` PR to pass-through dataframes, there was discussion about this as well: https://github.com/scikit-learn/scikit-learn/pull/11043 (currently, the FunctionTransformer cannot do both validation and pass-through as a DataFrame).

Related issue: https://github.com/scikit-learn/scikit-learn/issues/11447 about avoiding a `np.asarray` call in `check_array` for duck-typed arrays (eg to preserve dask arrays instead of converting them to numpy arrays). There is certainly overlap with this issue, but since the heterogeneous data also occur in some other use cases, opening a separate issue. 
But @jnothman mentioned in that issue the idea to have something like `accept={'array','array-like','frame-like'}` in `check_array`.

----

So from the above use cases, some possible improvements for `check_array` to better handle object / heterogeneous data:

* Ability to pass through DataFrames as-is (as DataFrames, but potentially still check dtypes, missing values, etc)
  * Have a `accept_frame=True/False` like there is a `accept_sparse` ? Or have a more general `accept=` keyword that can be extended later (eg `accept=['array', 'frame']`)
* A way to indicate that the dtype should be "numeric-or-object". Meaning: if it is numeric, keep numeric, if it is object keep object (up to here it would be 'dtype=None' behaviour), but if no dtype yet, the coerced array should be either numeric or object). Basically ensuring a lists gets converted to object array unless it is fully numerical.
  * A new value that can be passed to `dtype` ? `check_array(X, dtype="numeric-or-object")`? Or making sure that something like `dtype=['numeric', 'object]` would work (although this would not preserve a string dtype).
* Follow the `force_all_finite` keyword also for object dtype data (until now `check_array` did not need to handle with object dtype data, so it could assume that only float data could ever contain NaNs.



user: jorisvandenbossche
issue_12137: state: open
title: AdaBoost feature_importances_ should support base estimators with coef_
body: Could AdaBoost feature_importances_ attribute support base estimators which have coef_ available?
user: hermidalc
labels: Easy: Well-defined and straightforward way to resolve
issue_12099: state: open
title: Gaussian Mixture Model Removing Components by giving means of (0,0) and variances of 0
body: When using a warm start GMM over several frames of video data, the GMM collapses some components to (0,0) and their covariances matrix to diagonal matrices of near zero values. I've created a github repo with a notebook reproducing the issue which can be loaded into a Binder using the button on the README.

[Reproduction Notebook](https://github.com/Adurden/GMM_Issue)


user: Adurden
issue_12080: state: open
title: new feature: add LOBPCG solver to Truncated PCA
body: #### Description
The code [https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/decomposition/truncated_svd.py](url) currently supports only

> algorithm : string, default = "randomized"
        SVD solver to use. Either "arpack" for the ARPACK wrapper in SciPy
        (scipy.sparse.linalg.svds), or "randomized" for the randomized
        algorithm due to Halko (2009).

while LOBPCG https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.lobpcg.html is already used in [http://scikit-learn.org/stable/modules/generated/sklearn.manifold.SpectralEmbedding.html](url) and [http://scikit-learn.org/stable/modules/generated/sklearn.cluster.spectral_clustering.html](url)

lobpcg_svd solver has been added in https://github.com/scikit-learn/scikit-learn/issues/12079 

LOBPCG solver added to Truncated PCA in 
https://github.com/lobpcg/scikit-learn/commit/ec61eef358cc8b9b6553cdeedd6b809246e9c716

#### Steps/Code to Reproduce

    from sklearn.decomposition import TruncatedSVD
    from sklearn.random_projection import sparse_random_matrix
    X = sparse_random_matrix(100, 100, density=0.01, random_state=42)
    svd = TruncatedSVD(n_components=5, n_iter=7, random_state=42)
    svd.fit(X)
    print(svd.explained_variance_ratio_)
    print(svd.explained_variance_ratio_.sum())
    print(svd.singular_values_)
    svd = TruncatedSVD(algorithm='lobpcg', n_components=5, n_iter=7, random_state=42)
    svd.fit(X)
    print(svd.explained_variance_ratio_)
    print(svd.explained_variance_ratio_.sum())
    print(svd.singular_values_)

#### Expected Results
LOBPCG is expected to outperform both ARPACK  and  "randomized" solvers for Truncated PCA for large problems, e.g., see comments at https://www.mathworks.com/matlabcentral/fileexchange/48-lobpcg-m 

#### Actual Results
Not supported

#### Versions
All

user: lobpcg
issue_12076: state: open
title: StratifiedGroupShuffleSplit
body: It would be nice to have a manner to perform GroupShuffleSplit with some level of stratification. 
Using [GroupShuffleSplit](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupShuffleSplit.html) does not prevent that a target (for classification problems) will be present just in train or test sets. 
user: paulaceccon
labels: Needs Decision: Requires decision
issue_12055: state: open
title: Mixture of Bernoulli distributions
body: #### Description

Is there some intention to implement a mixture of Bernoulli distributions? Could be a great feature to perform anomaly detection with discrete quantities as well. 

If you are not aversed to this idea I would like to implement it with EM. 

What do you think?





user: clumsyfox-z9o9z
issue_12006: state: open
title: Inconsistency between predict and predict_proba parameters in Pipeline
body: Pull request [#9304](https://github.com/scikit-learn/scikit-learn/pull/9304) fixed issue [#9293](https://github.com/scikit-learn/scikit-learn/issues/9293) by adding `**predict_params` to `predict` method of Pipeline, but `predict_proba` was not changed and still only takes one parameter. I don't see why `predict_proba` (and also `predict_log_proba`) shouldn't also be updated in the same fashion, but I may be missing something.
Also as a related change, `fit_predict` also has this inconsistency, but in this case it's harder to do as cleanly (as there is already `**fit_params` passed as `kwargs`) - but maybe adding `predict_params` just as a normal, optional dict would be justified?
user: Noctiphobia
issue_11993: state: open
title: Feature request: Random Forest node splitting sample size
body: During the development of a model, the faster one can iterate, the better. That means getting your training time down to a manageable level, albeit at the cost of some accuracy.  With large training sets, training time for random forests can be an impediment to rapid iteration. @jph00 showed me this awesome trick that convinces scikit-learn's RF implementation to use a subsample of the full training data set when selecting split variables and values at each node:

```python
def set_RF_sample_size(n):
    forest._generate_sample_indices = \
        (lambda rs, n_samples: forest.check_random_state(rs).randint(0, n_samples,n))

def reset_RF_sample_size():
    forest._generate_sample_indices = (lambda rs, n_samples:
        forest.check_random_state(rs).randint(0, n_samples, n_samples))
```

We both find that splitting using say 20,000 samples rather than the full 400,000 samples gives nearly the same results but with much faster training time. Naturally it requires a bit of experimentation for each data set to find a suitable subsample size, but in general we find that feature engineering and other preparation work is totally fine using a subsample. We then reset the sample size for the final model.

We appreciate your consideration of this feature request!
user: parrt
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_11915: state: open
title: [MRG] Add feature weight to isolation forest
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #11835 

#### What does this implement/fix? Explain your changes.
Add feature weight to isolation forest.

<!--
#### Any other comments?

Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: zjpoh
issue_11909: state: open
title: Request: element removal option for neighbors.KDTree
body: It would be useful to be able to delete elements from a previously constructed KD-tree. This would be a method of the KDTree class, i.e. KDTree.delete(index). This could save time in constructing a new KD tree in the case of, for example, decimation.

This is outlined in the original <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.160.335&rep=rep1&type=pdf">KD-tree paper (Bentley)</a> in section 5.
user: mwharton3
labels: Needs Decision: Requires decision
issue_11897: state: open
title: Remove Hardcoded Thresholds in Repr
body: #### Description
Repr cuts off outputs when they are going to be longer than 500 characters.

While the [python docs ](https://docs.python.org/3/library/functions.html#repr) dont say it is a requirement, they do state "For many types, this function makes an attempt to return a string that would yield an object with the same value when passed to eval"

It would be nice to be able to do something similar to numpy, where we can set the threshold ourselves, so if desired repr can be used with eval to reconstruct an object. For reference, this is how np allows users to set it

```
np.set_printoptions(threshold=np.inf)
```


#### Steps/Code to Reproduce

Calling repr on any objects with a large number of params

#### Expected Results
Full output of below (without the ...)
#### Actual Results
```
Pipeline(memory=None,
     steps=[('minmaxscaler', MinMaxScaler(copy=True, feature_range=(0, 1))), ('selectfrommodel', SelectFromModel(estimator=LogisticRegression(C=100.0, class_weight=None, dual=False, fit_intercept=True,
          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
          penalty='l2', rando...n_jobs=1,
            oob_score=False, random_state=None, verbose=0,
            warm_start=False))])
```

_Note the ... after random_
#### Versions

All
user: benjaminpatrickevans
labels: Needs Decision - API: Requires decision regarding API
issue_11885: state: open
title: Common tests: no print to stdout?
body: I just saw that #10365 added a verbose=1 default argument and I don't think we have tests against that. I think we should be quite by default, but not sure we want to enforce that in ``check_estimator``?

Generally we might want to be a bit more lenient with some things in ``check_estimator`` than we are internally, I think (say error messages), but not sure yet how to best do that.
user: amueller
labels: module:test-suite: everything related to our tests
issue_11865: state: open
title: RFC LogisticRegression code is way to opaque
body: After the release we should refactor the logistic regression code.
Right now, path computation, optimizers and OVR are all a tangled mess.
user: amueller
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_11863: state: open
title: has_fit_parameter() can fail with meta-estimators
body: #### Description
When an `AdaBoostClassifier` (and presumably other meta-estimators relying on sample weighting) has a `GridSearchCV` classifier as its base estimator, it cannot be fitted because `GridSearchCV` fails the `has_fit_parameter` check for `sample_weight`, even though its own underlying estimator might support `sample_weight`.

#### Steps/Code to Reproduce
```python
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
base_estimator = GridSearchCV(DecisionTreeClassifier(),
                              param_grid=dict(max_depth=[1, 2, 3]))
estimator = AdaBoostClassifier(base_estimator)

# Fitting fails because GridSearchCV doesn't have a sample_weight fit parameter,
# although the underlying estimator (DecisionTreeClassifier) does.
estimator.fit(X, y)
```

#### Expected Results

Ideally, `has_fit_parameter` would check the underlying estimator of a meta-estimator for fit parameters.

#### Actual Results

A `ValueError` is raised:

```python
Traceback (most recent call last):
...
ValueError: GridSearchCV doesn't support sample_weight.
```
<!-- Please paste or specifically describe the actual output or traceback. -->

#### Versions

Darwin-17.7.0-x86_64-i386-64bit
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24) 
[Clang 6.0 (clang-600.0.57)]
NumPy 1.15.0
SciPy 1.1.0
Scikit-Learn 0.20.dev0
user: artemmavrin
labels: Needs Decision: Requires decision
issue_11861: state: open
title: Random Forest Feature Contributions for Individual Predictions
body: #### Description

"Feature contributions" meaning, the effects of each feature's value on a given prediction. This concept is explained well in [this paper](http://eprints.whiterose.ac.uk/79160/1/feature_contribution_camera_ready.pdf). I've written code that uses the method described in section 3 of that paper, moving down the path of the tree to get these "feature contributions" in each tree as the paper describes, then averaging them over all trees.

The code I've written calculates this at the same time that it does the actual predictions (i.e. in the same "while child node is not a leaf" loop) and returns both. We need to follow the sample's path down the tree to calculate the prediction anyway, so it was much faster to do this at the same time that the contributions were calculated rather than moving down the same path of the tree two separate times, once for the prediction and once for the contributions.

So I have a method called something like "predict_with_contributions". I'd also be open to splitting off the feature contributions calculation into it's own method if having both in the same method sounds too messy/confusing. If I were to do this, it might make sense for the code to use the get_decision_path code from this PR: #5487
user: shkupfer
issue_11843: state: open
title: Cannot distinguish missing and zero valued features ``load_svmlight_file`` 
body: #### Description
The LibSVM format (also SVM-Light) is commonly used to store datasets. It was originally devised for SVMs which cannot handle non-numerical data or missing data. The data format is as follows

`` <target_value> 1:<feature_1_val> 2:<feature_2_val>  ... ``
The format can be used to handle sparse data-sets - if a feature is not included, when loaded into a sparse matrix, it will be assumed to be zero. 

The function ``sklearn.datasets.load_svmlight_file`` takes a file in this format and converts it into a sparse matrix (``scipy.sparse.csr_matrix``). This can pose a problem with techniques which can handle missing data or in situations where the absence of data is important i.e. the data is not missing completely at random or there are features for which a zero value is different to the data being missing.  For example, if imputation is being used, it is important that the values which are genuinely zero are not being imputed.  

Whilst this may be a limited use case, I propose modifying the function ``sklearn.datasets.load_svmlight_file`` to include an additional option: ``missing_data_value`` which will be the value that all missing data takes (e.g. ``np.NaN``). The default value should remain at zero to avoid breaking existing code. 

(I apologises if I have not followed the guidlines for reporting an issue properly - this is the first time I've made an issue - thank you for your understanding!).
user: MrinankSharma
issue_11835: state: open
title: High dimensional data support in IsolationForest by weighting attributes selection
body: #### Description
IsolationForest allows choosing weighted sampling of attributes when building isolation trees. Adding this part to the algorithm can help to avoid noise attributes or redundant features. 

#### Steps/Code to Reproduce
```python
# first try, only relevant features
X = np.random.exponential(size=(5000,2))
clf = IsolationForest(max_features=1.0,
                      max_samples=2**8,
                      n_estimators=200,
                      n_jobs=-1)
# X = np.hstack([X, np.random.random(size=(len(X), 10))])
clf.fit(X)
y_pred = clf.decision_function(X)
print np.histogram(y_pred)[0]

# second try, with irrelevant features
X = np.random.exponential(size=(5000,2))
X = np.hstack([X, np.random.random(size=(len(X), 10))])
clf = IsolationForest(max_features=0.5,
                      max_samples=2**8,
                      n_estimators=200,
                      n_jobs=-1)
clf.fit(X)
y_pred = clf.decision_function(X)
print np.histogram(y_pred)[0]
```

#### Expected Results

I expected to get similar prediction histograms, because this algorithm should be relatively robust to this type of noising. The features (or "attributes" in the article) to use when building Isolation Trees should be chosen randomly, but with weights according to some test, Kurtosis for example.

See section 5.6 in the work "Isolation-based Anomaly Detection" by Fei Tony Liu, Kai Ming Ting and Zhi-Hua Zhou [https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/tkdd11.pdf](url)

#### Actual Results
The second histogram looks like results with completely uniform distributed data.
**Output:**
```
[   3   26   50   80   97  167  322  625 1384 2246]
[  13   39  175  473  931 1356 1195  641  167   10]
```
user: elado-alon
issue_11824: state: open
title: fetch_openml: Maybe download task information as well as dataset
body: OpenML.org defines [tasks](https://docs.openml.org/#tasks), associated with each dataset, which define a cross-validation procedure and an evaluation metric. I think we should consider fetching task information along with the data, and probably construct a cross validation splitter (though I'm not sure how we make this be identical to the procedure run on OpenML.org). I'm not sure what we can do with the evaluation metric information... we can construct a scorer only in some cases.
user: jnothman
labels: Moderate: Anything that requires some knowledge of conventions and best practices
Needs Decision: Requires decision
issue_11820: state: open
title: fetch_openml: Return a numeric target for multilabel problems
body: Scikit-learn does not currently support multilabel (or multioutput multiclass) targets in its classifiers, where the labels/classes are represented as strings. `fetch_openml` currently returns string-valued targets for multilabel datasets. As such we use `yeast.target == 'TRUE'` in [`plot_classifier_chain_yeast.py`](https://github.com/scikit-learn/scikit-learn/blob/master/examples/multioutput/plot_classifier_chain_yeast.py) to convert a string array into a binary numeric array before passing to a classifier.

We should not have to do this, at least for multilabel problems. Perhaps we can only do this automatically when the intended positive class is obvious, e.g. when the string values are 'FALSE' and 'TRUE'. In this case, we would automatically encode the target as integers.

@janvanrijn: is use of {FALSE, TRUE} as the nominal values in multilabel problems a standard used in OpenML, or just something that happened to be the case in `yeast`.
user: jnothman
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_11784: state: open
title: Label Spreading: knn Kernel Variant differs from Reference Paper
body: According to [label_propagation.py#L487](semi_supervised/label_propagation.py#L487), the reference paper for the Label Spreading implementation is [Zhou et al.: Learning with local and global consistency (2004)](http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.115.3219). 

But there are two things not in line with the paper when using the *knn* kernel variant for Label Spreading:

#### (A) Self-loops in affinity matrix
According to the paper, the adjacency matrix should have **no self-loops** (W_ii = 0). But when taking a look on the relevant line in [_get_kernel, L134](https://github.com/scikit-learn/scikit-learn/blob/f0ab589f/sklearn/semi_supervised/label_propagation.py#L134) (called with `y=None`, see [L516](semi_supervised/label_propagation.py#L516) for Label Spreading and [L397](semi_supervised/label_propagation.py#L397) for Label Propagation, `kneighbors_graph` get's called with `mode='connectivity'`, which results in `include_self=True`, according to the [docs](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.kneighbors_graph.html). 
Therefore, a more correct way would be calling it with `include_self=False`. 

Note that this change would not only modify the kneighbors_graph affinity matrix used for Label Spreading, but also the one for Label Propagation. When looking at the reference paper for Label Propagation, [Zhu et al.: Learning from labeled and unlabeled data with label propagation (2002)](http://pages.cs.wisc.edu/~jerryzhu/pub/CMU-CALD-02-107.pdf), they seem not to require that there should be no-self loops. 
But in sklearn's other reference, [Bengio et al.: In Semi-Supervised Learning (2006)](https://www.researchgate.net/publication/238675708_Label_Propagation_and_Quadratic_Criterion), mentioned [here](semi_supervised/label_propagation.py#L48), they advocate forcing W_ii=0 anyways - so I would use `include_self=False` for Label Propagation's kneighbors_graph as well.

#### (B) Laplacian Matrix Calculation
In [label_propagation#L517](https://github.com/scikit-learn/scikit-learn/blob/f0ab589f/sklearn/semi_supervised/label_propagation.py#L517), we see that `sparse.csgraph.laplacian` gets called. The default value for this method's `use_out_degree` is `False`. This is not a problem if the affinity matrix is symmetric, but as knn graphs usually are **not symmetric**, `use_out_degree=True` should be used. 
One can better see this issue when getting the diagonal via `sparse.csgraph.laplacian(..., normed=False, return_diag=True)` - when `use_out_degree=False`, the diagonal values won't resemble the `k` from `kneighbors_graph`.

### Summary
To conform with the reference paper, we should
- In [label_propagation#L517](https://github.com/scikit-learn/scikit-learn/blob/f0ab589f/sklearn/semi_supervised/label_propagation.py#L517), call `kneighbors_graph` with `include_self=False`
- In [label_propagation#L134](https://github.com/scikit-learn/scikit-learn/blob/f0ab589f/sklearn/semi_supervised/label_propagation.py#L134), call `sparse.csgraph.laplacian` with `use_out_degree=True`

If you agree with me, I can submit this small PR and fix the tests, in case one breaks. 😊
user: markusdosch
issue_11778: state: open
title: Consensus Index for Clustering Model Selection
body: The sentence:
```
XXX can also be useful in a purely unsupervised setting as a
  building block for a Consensus Index that can be used for clustering
  model selection (TODO).
```
is in the description of every clustering metric that requires a ground truth label. Is there a write-up somewhere of an example workflow to build such a Consensus Index using these metrics? I searched through old issues and the original PRs creating this documentation and I can't see anything that looks like that. Though to be fair, I'm a big dummy, so I could just be missing it. If there isn't, is this TODO still really a TODO and is there an existing issue for it that I'm also too dumb to find? Thanks! 

❤️ 

cc @ogrisel since you seem to be the og implementor and the original author of these docs.
user: robbymeals
issue_11777: state: open
title: Unavoidable "y_true and y_pred contain different number of classes" error inside a CV loop
body: #### Description
During cross-validation on a multi-class problem, it's technically possible to have classes present in the test data that don't appear in the training data.

#### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.metrics import make_scorer, log_loss
from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold
from sklearn.naive_bayes import BernoulliNB

rs = np.random.RandomState(1389057)

y = [
    'cow',
    'hedgehog',
    'fox',
    'fox',
    'hedgehog',
    'fox',
    'hedgehog',
    'cow',
    'cow',
    'fox'
]

x = rs.normal([0, 0], [1, 1], size=(len(y), 2))

model = BernoulliNB()

cv = StratifiedKFold(4, shuffle=True, random_state=rs)

param_dist = {
    'alpha': np.logspace(np.log(0.1), np.log(1), 20)
}

search = RandomizedSearchCV(model, param_dist, 5,
                            scoring=make_scorer(log_loss, needs_proba=True), cv=cv)

search.fit(x, y)
```

#### Expected Results

Either:
1. Predicted classes from `predict_proba` are aligned with classes in the full training data, not just the in-fold subset.
2. Classes not in the training data are ignored in the test data.

#### Actual Results

Predicted classes from `predict_proba` are aligned with classes in the in-fold subset only, but classes not in the training data are still used in the test data, causing the error.

I understand that this is normatively "correct" behavior, but it makes it hard/impossible to use in cross-validation with the existing APIs.

From my perspective, the best solution would be to have `RandomizedSearchCV` pass a `labels=self.classes_` argument to its scorer. I'm not sure how well that generalizes.

#### Versions

```
Linux-3.10.0-514.26.2.el7.x86_64-x86_64-with-redhat-7.3-Maipo
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51) [GCC 7.2.0]
NumPy 1.15.0
SciPy 1.1.0
Scikit-Learn 0.19.1
```

user: gwerbin
issue_11759: state: open
title: SGDClassifier: call to partial_fit classes should include all valid labels that can be in y
body: I'm incrementally training SGDClassifier for multiclass classification using partial_fit. Here is part of my code.

```
from sklearn.datasets import load_svmlight_file
import numpy as np
from scipy.sparse import hstack
from sklearn.linear_model import SGDClassifier
import math

data_path = "/home/chandresh/ckm/data/multiclass data/"
tranfile= 'letter.scale.tr' # download this data from 
# https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass.html

def get_data():
    data = load_svmlight_file(data_path+tranfile)
    return data[0], data[1].astype(int)

X, y = get_data()

# add bias column

X = hstack([np.array([[1] for _ in range(X.shape[0])]), X])
X = X.toarray()

n, d = X.shape
no_classses =len( np.unique(y)) # 26 classes


clf = SGDClassifier(loss='log')
# for online learning, I'm extracting at least one instance from each class and doing partial_fit
# xtr, ytr contain one instance from each class.
xtr=[]
ytr=[]
ind=[]

for i in range(1,no_classses+1):
    s, = np.where(y==i)
    ind.append(s[0])

xtr = X[ind,:]
ytr = y[ind]

X = np.delete(X,ind,0)
y = np.delete(y, ind,0)

n = X.shape[0]
classes  = np.unique(y)
clf.partial_fit(xtr,ytr, classes)
for t in range(0,n):
    xt = X[t,:]
    yt = int(y[t])-1
    p  = clf.predict_proba([xt])      
     # update the MLP
    clf = clf.partial_fit([xt],[yt], classes)  
```
Error came during `class_weight` computation. I don't understand, why ` compute_class_weight` is called at all though I'm passing `class_weight=None` by default. @jnothman any idea?
user: chandreshiit
labels: Needs Investigation: Issue requires investigation
issue_11749: state: open
title: true positive/negative rate at a certain false positive/negative rate and vice versa
body: I am wondering if it would be a useful feature for the library to include a classification metric given a fixed other classification metric. For example, in my past research, we were interested in the true-positive rate at a certain, fixed (and often small) false-positive rate, i.e. the ROC value on the very left of the scale.
user: FabianFalck
issue_11723: state: open
title: [MRG] MLP - add class weights support
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
Fixes #9113 
(should close stalled pull request #10588)

#### What does this implement/fix? Explain your changes.

- [x] Add sample_weight
  - [x] Add support to multiply sample weight sample wise on loss functions:
    - [x] log_loss
    - [x] binary_log_loss
    - [x] squared_loss
  - [x] Add sample_weight parameter to fit and partial_fit
    - [x] Propagate to relevant functions
    - [x] Validate sample_weight size on fit
  - [x] Add tests
    - [x] Test directly loss functions sample_weight invariance 
    - [x] Test fit validation of sample_weight size

- [x]  Add class_weight support
   - [x] Add support on fit and partial fit
   - [x] Add test for behaviour correctness 

- [ ] Test with `utils.estimator_checks.check_class_weight_classifiers`: Currently (Nov 18) have issues passing this test, however succeeded when using one neuron MLP
#### Any other comments?

I started by adding sample_weight parameter support, next should add class_weight.
Please share your comments

Thanks

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: dorcoh
issue_11715: state: open
title: intercept_scaling should be set explicitly
body: I think we should warn the user when liblinear is used (in LinearSVC/R or LogisticRegression) and fit_intercept is used, if they do not explicitly set intercept_scaling. This is to improve awareness of liblinear's failure to regularise the intercept.

I would propose:
* Set the default intercept_scaling to 'warn'
* Raise a UserWarning if `fit_intercept and intercept_scaling == 'warn'`
user: jnothman
labels: Needs Decision: Requires decision
issue_11672: state: open
title: Q: Is changing ridge solver to sag for sparse intercept a good idea?
body: I noticed from one of the error messages that we're automatically changing the ridge solver to SAG if the data is sparse and fit_intercept is True.
Has anyone checked how well that works if the data is not appropriately scaled? When I checked for logistic regression, SAG wasn't doing so well unless the data is scaled.

We could possibly just add to the warning that the data should be scaled, or assert that the data is somewhat scaled, or we could error and tell the user to change the solver and scale the data.

It's a bit of a strange behavior actually, as the user explicitly specified a solver, but then the solver is changed in spite of that.

So two-part question:
- should we change the solver if the user explicitly specified a different solver?
- Is moving to SAG automatically (even for 'auto') a good choice?
user: amueller
issue_11671: state: open
title: [MRG] Add Penalty factors for each coefficient in enet ( see #11566)
body: 
This pull request implements the feature in issue #11566 

Also, I added a test for this feature.


user: doaa-altarawy
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_11669: state: open
title: Use Fisher's exact test with the function SelectKBest
body: Is there a way to implement Fisher's exact test in Scikit-learn to use it in an optimal way with SelectKBest ?
user: AlexandraBomane
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_11668: state: open
title: roc_auc_score min_tpr parameter?
body: We recently added to the roc_auc_score the ability to only get the area under a section of the curve below a maximum false positive rate (`max_fpr`). Does it also make sense to constrain with a minimum true positive rate (`min_tpr`) or equivalently `max_fnr`?

I also note that in the calibration tool proposed in #10117, methods are  `max_tpr` and `max_tnr`.
user: jnothman
issue_11649: state: open
title: Fix Tree Median Calculation for MAE criterion
body: `K` is not being updated properly in certain situations where we have non-uniform sample weights.  This occurs during a removal/pop or push onto the `WeightedMedianCalculator`.  The proposed fix solves this problem by identifying the exact index a new / old sample is added / removed and applying additional logic to update K correctly.

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #10725 (BUG Median not always being calculated correctly for DecisionTrees in the WeightedMedianCalculator)

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: JohnStott
issue_11639: state: open
title: [MRG] ENH: Adds inverse_transform to ColumnTransformer
body: #### Reference Issues/PRs
Fixes #11463

#### What does this implement/fix? Explain your changes.
1. Running `inverse_transform` with overlap or `drop` will raise a `ValueError`
2. `_calculate_inverse_indices` is used to connect indices from the output space back to the input space.
user: thomasjpfan
labels: Needs Decision: Requires decision
issue_11633: state: open
title: Is utils/__init__.py the right place for what's inside ?
body: The content of the file `utils/__init__.py` does not seem appropriate.
I mean it works, but it does not seem to be good practice.

It contains a bunch of functions and classes which can be used in many places in sklearn.
Those functions should not be in a `__init__.py`, but in a submodule instead.
Moreover when defining such functions, contributors often don't know where to put them cause this is not an intuitive place for that.

If I understand correctly those functions does not fit in any of the existing submodules of 'utils' and have very different purposes. So I propose to make a new submodule, `misc.py` or `utils.py` or whatever name you think is best, and put all the orphan functions in there.

Note: The same process could be applied to 
`utils/sparsetools/__init__.py`
`_build_utils/__init__.py`
`__check_build/__init__.py`

Is this something you could consider ?
user: jeremiedbb
labels: Needs Decision: Requires decision
issue_11601: state: open
title: [MRG] Issue 9698 safe and short repr
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Contains the basis for fixing #9698.

#### What does this implement/fix? Explain your changes.

Utility functions that provide safe and friendlier versions of str() and repr() were added to `utils`: `safe_str()` and `safe_repr()`.

These functions _should never crash_ and should _always return a string_.

They are intended to be used in `raise` statements and warning calls. I created a **list that should contain most (if not all) such places**: https://gist.github.com/lebigot/8526af2425355b5cf837068c4182fc2d.

I have thus started to go through this list (in the listed order): this pull requests contains the first changes.

#### Any other comments?

I considered that there is no need to protect the representation of something like `arr.shape` when `arr` is supposed to be a NumPy array or equivalent (and has likely be used as such before the error message).

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->
user: lebigot
issue_11589: state: open
title: Allowing sparse y in GridSearchCV and cross_val_score
body: This PR addresses #8438 (stalled PR which aimed at fixing #4225).

The changes in this PR aim at allowing `GridSearchCV` and `cross_val_score` to accept sparse `y`. This can be particularly useful in some extreme learning situations where one has a lot of samples and many labels. This is only useful as long as `GridSearchCV` is passed an estimator which also accepts a sparse `y`. Future PRs such as #4354 are needed to add support for sparse y in Scikit-Learn estimators.

Details:

- in `model_selection/_search.py` and `model_selection/tests/test_search.py`: allow GridSearchCV to be passed a sparse y. The `fit` method of the `BaseSearchCV` class calls `check_cv`, in which `type_of_target` needed to support sparse input. 
- in `model_selection/_split.py` and `model_selection/tests/test_split.py`: The `split` method of CV classes (`KFold`, `StratifiedKFold`,...) needed to accept a sparse `y`. The typical use case of sparse `y` is multi-label. Therefore, `StratifiedKFold` and `StratifiedShuffleSplit` should not be used, unless `y` has shape `(n_samples,)` or `(n_samples, 1)`. In this case, the sparse `y` is converted to a dense array.
- in `utils/multiclass.py` and `utils/test/test_multiclass.py`: allow `type_of_target` and `unique_labels` to accept a sparse matrix.
- in `/utils/sparsefuncs.py` and `/utils/tests/test_sparsefuncs.py`: added a function to obtain the unique elements in a sparse array.
- in `model_selection/_validation.py` and `model_selection/tests/test_validation.py`: allow `cross_val_score` to accept a sparse `y`.

@GaelVaroquaux, @jnothman  

user: jbschiratti
issue_11579: state: open
title: Memoize tokenisation in CountVectorizer
body: IIRC, previous attempts to improve efficiency in CountVectorizer have found that tokenization is the primary bottleneck. In my private extensions, memoizing the mapping (`s -> tokenize(preprocess(s))`) can improve runtime greatly, especially when the vectorizer is in a cross-validation pipeline (which ColumnTransformer etc. helps encourage).

Challenges to memoization:
* it might makes sense to use in-memory caching rather than on-disk given the relatively small blobs being cached.
* it has to be conditioned on all relevant constructor parameters to the CountVectorizer (i.e. anything that goes into build_analyzer up to and including tokenization)
user: jnothman
issue_11566: state: open
title: Add Penalty factors for each coefficient in enet (similar to the R's glmnet library)
body: #### Description

This is a feature request in allow flexibility in elastic net, which allows the user to apply separate penalties to each coefficient of the L1 term. The default value of this penalty factor is 1, meaning the regular elastic net behavior. If the penalty factor of a feature is Zero, it means it's not penalized at all, and that the user wants this feature to be always there in the model.

This feature is very useful in Bioinformatics and systems biology (that's why it is in Stanford's R package glmnet). With this feature, the user can run feature selection on a set of genes while making sure some genes stay in the system and not penalized (because of prior knowledge that they are involvid the system).

Here is glmnet documentation explaining the penalty factor. Mainly, it's controlling the selection weight on the lasso term:

[https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html#lin](https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html#lin)


This feature is used in several papers, I've implemented in it in one of mine in scikit-learn. I've requests from other biologists to use this method in Python without having to recompile scikit-learn. I can do a pull request with this feature.

Papers using this feature:

- Altarawy, Doaa, Fatma-Elzahraa Eid, and Lenwood S. Heath. "PEAK: Integrating Curated and Noisy Prior Knowledge in Gene Regulatory Network Inference." Journal of Computational Biology 24.9 (2017): 863-873.
- Greenfield, Alex, Christoph Hafemeister, and Richard Bonneau. "Robust data-driven incorporation of prior knowledge into the inference of dynamic regulatory networks." Bioinformatics 29.8 (2013): 1060-1067.
- Friedman, Jerome, Trevor Hastie, and Rob Tibshirani. "Regularization paths for generalized linear models via coordinate descent." Journal of statistical software 33.1 (2010): 1.
user: doaa-altarawy
labels: Moderate: Anything that requires some knowledge of conventions and best practices
Needs Decision - API: Requires decision regarding API
issue_11549: state: open
title: [WIP] use more robust mean online computation in StandardScaler
body: #### Reference Issues/PRs

Closes #5602

#### What does this implement/fix? Explain your changes.

use parallel algo from https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online_algorithm

I still new to check if same problem occurs on sparse matrices and for variance... which I suspect

user: agramfort
issue_11543: state: open
title: Support load_svmlight_file in PyPy
body: At present, `load_svmlight_file` is not supported with PyPy because it relies on interacting with array.array from Cython which is not supported and probably won't be in the near future.

See https://github.com/scikit-learn/scikit-learn/issues/11540 for more details and possible workarounds.
user: rth
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_11536: state: open
title: Liblinear convergence failure everywhere?
body: Recently we made liblinear report convergence failures.
Now this is reported in lots of places. I expect our users will start to see it everywhere. Should we change something? It's weird if most uses of LinearSVC result in a "failure" now.
user: amueller
labels: Needs Decision: Requires decision
issue_11528: state: open
title: Output dimension consistency between PCA and LinearDiscriminantAnalysis
body: #### Description
This follows #11526, regarding some sometimes unexpected behaviour: that LDA can sometimes truncate the `scalings_` (and the output of transform), overwriding in a sense what the user has specified in `n_components`. This time this tackles the case where it happens because the input is made of collinear vectors. The behaviour of `PCA` and `LinearDiscriminantAnalysis` is not the same regarding collinear input: PCA (and all other `sklearn.decomposition` algorithms) transforms the input with some always zero features, but raises no Warning, whereas LDA (with the default solver 'svd') truncates the transform output but raises `"Variables are collinear"`. I wondered if it wouldn't better to uniformize a bit the behaviour, by for instance: 
- making PCA raise a warning in that case
- and making LDA transform return a transformation including the zeros (not truncating)
Also maybe the warning "Variables are collinear" could be improved by saying that some dimensions of the output will be zeros. 

#### Steps/Code to Reproduce
Here is an example that shows the different behaviours from LDA and PCA:
Example:
```python
import numpy as np
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.decomposition import PCA
basis = np.random.randn(3, 5)  # a basis of rank 3 to generate other vectors
X = np.random.randn(10, 3).dot(basis)  # we create 10 collinear points, of n_features=5 and rank is 3
y = np.hstack([1, 1, 2, 2, 3, 3, 4, 4, 5, 6])  # we create a lot of classes so the truncation is due to collinearity and not number of classes
for model, model_name in zip([PCA(), LinearDiscriminantAnalysis()], ['PCA', 'LDA']):
	model.set_params(n_components=5)
	print('-'*40 + model_name + '-'*40)
	model.fit(X, y)
	print('transform_shape: %s' % str(model.transform(X).shape))
	print((model.transform(X)**2).sum(axis=0)) # sum of squares of features to see features always zero
	# print(model.transform(X))
```

#### Expected Results
something like that: 
```pytb
----------------------------------------PCA----------------------------------------
transform_shape: (10, 5)  
 warnings.warn("Variables are collinear. Two features of the output space will always be zero.")
[notzero notzero notzero zero zero]
----------------------------------------LDA----------------------------------------
/home/will/Code/sklearn-forks/wdevazelhes/scikit-learn/sklearn/discriminant_analysis.py:388: UserWarning: Variables are collinear.
  warnings.warn("Variables are collinear. Two features of the output space will always be zero.")
transform_shape: (10, 5)
[notzero  notzero  notzero zero zero]
```
#### Actual Results
```pytb
----------------------------------------PCA----------------------------------------
transform_shape: (10, 5)
[9.32882671e+01 1.86702898e+01 6.05330823e+00 1.30856483e-30
 2.75278672e-30]
----------------------------------------LDA----------------------------------------
/home/will/Code/sklearn-forks/wdevazelhes/scikit-learn/sklearn/discriminant_analysis.py:388: UserWarning: Variables are collinear.
  warnings.warn("Variables are collinear.")
transform_shape: (10, 3)
[13.9232471   8.3425689   5.96980738]
```

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
Linux-4.4.0-130-generic-x86_64-with-debian-stretch-sid
Python 3.6.3 |Anaconda, Inc.| (default, Nov  3 2017, 19:19:16) 
[GCC 7.2.0]
NumPy 1.14.3
SciPy 1.1.0
Scikit-Learn 0.20.dev0



<!-- Thanks for contributing! -->

user: wdevazelhes
issue_11512: state: open
title: Extracting explained variance in SparsePCA
body: Why is there no attribute called "explained_variance_ratio_" in Sparse PCA as in conventional PCA with scikit? 

Is there an alternative way to extract this?

Thank you.
user: kimrants
labels: Needs Decision: Requires decision
issue_11499: state: open
title: Enable IP Anonymization for the Google Analytics tracker of scikit-learn.org
body: This would require upgrading the tracker code in `doc/themes/scikit-learn/layout.html` to use the new `analytics.js` code and then configure use `ga('set', 'anonymizeIp', true);` as explained in:

https://developers.google.com/analytics/devguides/collection/analyticsjs/ip-anonymization

More details in:

https://support.google.com/analytics/answer/2763052?hl=en

The tricky part is that we also need to do that change for all the old versions of the scikit-learn documentation.

UPDATE: to close this we still need to edit the HTML file of pre-0.22 website archives: https://github.com/scikit-learn/scikit-learn/issues/11499#issuecomment-1039143833
user: ogrisel
issue_11493: state: open
title: Make kmeans++ subsample data for initialization on large n_samples or n_clusters
body: Discussed with @ogrisel yesterday that it would be nice to subsample the data before applying kmeans++ if the data is big, as otherwise the initialization takes longer than the clustering.
This needs some benchmarks but could drastically speed up the overall algorithm.
user: amueller
issue_11463: state: open
title: Support inverse_transform in ColumnTransformer
body: Since ColumnTransformer is primarily used to apply basic preprocessing, it would be extremely helpful if it (and `FeatureUnion` while we're at it) supported `inverse_transform` (where each of its constituent estimators does too). This also would replace some functionality we might otherwise support with `categorical_features` or `ignored_features` etc parameters (H/T @qinhanmin2014).

In order for it to support `inverse_transform`, it would need to have an index of which columns in the output came from which transformers, which is what I intended to solve in #1952. This is not hard as long as `fit_transform`, rather than `fit`, is called, or as long as all the constituent transformers support `get_feature_names`. Maybe we could force a run of `transform` when fit is called in order to check the output size.

(Alternatively, it might be nice to have a consistent API for a transformer to declare its output size, which `n_components_` does for a handful of estimators.)
user: jnothman
issue_11456: state: open
title: Treat Predict as a Numpy Generalized Ufunc
body: My understanding is that prediction is always a row-wise broadcastable operation, making it trivially parallelizable across parallel collections like dask.array/dask.dataframe.  We implement this in dask-ml with meta-estimators like ParallelPostFit

### Today

```python
# Train with local data
sgd = sklearn....SGDClassifier()
sgd.fit(X_numpy, y_numpy)

# Predict on larger data
dsgd = ParallelPostFit(sgd)
dsgd.predict(X_dask, y_dask)
```

Alternatively people might just use dask array options directly

```python
predictions = dask.array.map_blocks(sgd.predict, X_dask, y_dask)
```

### Tomorrow?

These workarounds are a little awkward and not a major pain point, but it still might be nice to support this kind of operation directly in scikit-learn

```python
# Train with local data
sgd = sklearn....SGDClassifier()
sgd.fit(X_numpy, y_numpy)

# Predict on larger data
sgd.predict(X_dask, y_dask)
```

### GUFunc protocols

We would want to avoid explicit `if isinstance(X, dask.Array)` checks within the Scikit-Learn codebase, so presumably we would want to find some sort of protocol that any sort of array library might be able to implement to inform scikit-learn that it is able to take over.  That protocol may already exist in Numpy's [GUFunc](https://docs.scipy.org/doc/numpy/reference/c-api.generalized-ufuncs.html) model.  If it was easy to make Scikit-Learn's `Estimator.predict` a Numpy GUFunc then we would get this for free today (Dask array supports gufuncs through the `__array_ufunc__` protocol (see [dask gufunc documentation](http://dask.pydata.org/en/latest/array-gufunc.html) for more information).

However, there are challenges to implementation.  My understanding is that it can be quite difficult to implement the GUFunc protocol today, but I wonder if this might be a nice use case to motivate future development on the Numpy side. (cc @stefanv @shoyer)
user: mrocklin
issue_11447: state: open
title: Avoid np.asarray call in check_array for duck-typed arrays
body: Would it be reasonable to avoid np.asarray calls in check_array if the input is array-like?

My guess is that the general answer is "No.  The np.asarray call is important to guarantee consistency within scikit-learn's estimators.  We strongly value consistency."

The context here is that after the recently merged https://github.com/scikit-learn/scikit-learn/pull/11308 , some scikit-learn transformers like RobustScalar that used to work on dask arrays no longer work because they auto-coerce their inputs into numpy arrays.  This likely comes up in other situations as well.

user: mrocklin
issue_11435: state: open
title: Metrics which are degenerate with a single sample should raise an error if passed a single sample
body: A metric such as `r2_score` requires comparing the distributions of the true and predicted values. With a single sample it either returns 1 for equality or 0 for inequality (which already is a bit weird given that it's a metric over continuous variables).

Instead, we should return NaN or issue a warning or raise an error. We should identify other metrics where this would also be appropriate behaviour.

Do others think this is the correct course of action?
user: jnothman
issue_11433: state: open
title: Add common tests for pairwise non-classifiers
body: #9701 introduced common tests for estimators taking a precomputed pairwise affinity/kernel or dissimilarity/distance matrix, but restricted its scope to classifiers. Certainly, these tests should be generalised to regressors, and probably to clusterers and transformers.

For instance, in all cases, the estimators should raise a good error if expecting precomputed input, but the training data is not square.
user: jnothman
labels: module:test-suite: everything related to our tests
issue_11430: state: open
title: [MRG] DOC add guideline for choosing a scoring function
body: Solves issue #10584.
Adds the new chapter "3.3.2. Which scoring function should I use?" to the user guide.
user: lorentzenchr
issue_11429: state: open
title: ClassifierChain does not support GroupKFold
body: ```python
from sklearn.datasets import make_multilabel_classification
from sklearn.multioutput import ClassifierChain
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupKFold

X, Y = make_multilabel_classification()
ClassifierChain(LogisticRegression(), 'random', cv=GroupKFold(3)).fit(X, Y)
# ValueError: The 'groups' parameter should not be None.
```
user: uatach
issue_11417: state: open
title: Graphical lasso Issue with graph_lasso() method
body: #### Description
 Dear All,
 I am working on replicating a paper titled “Improving Mean Variance Optimization through Sparse Hedging Restriction”. The authors’ idea is to use Graphical Lasso algorithm to infuse some bias in the estimation process of the inverse of the sample covariance matrix. The graphical lasso algorithm works perfectly fine in R, but when I use python on the same data with the same parameters I get two sorts of errors:
 
1-  If I use coordinate descent (cd ) mode as a solver, I get a floating point error saying that: the matrix is not symmetric positive definite and that the system is too ill-conditioned  for this solver. “FloatingPointError: Non SPD result: the system is too ill-conditioned for this solver. The system is too ill-conditioned for this solver”  (The thing that bugs me is that I tried this solver on a simulated Positive definite matrix and It game me this error)
 
2-  If I use the Least Angle Regression (LARS) mode (Which is less stable but recommended for ill-conditioned matrices) I get an  Overflow error stating that the integer is too large to be converted to a float “OverflowError: int too large to convert to float”
 
To my knowledge, unlike C++ and other languages, python is not restricted by an upper maximum for integer numbers (besides the capacity of the machine itself). Whereas the floats are restricted. I think this might be the source of the later problem. (I have also heard in the past that R is much more robust in terms of dealing ill-conditioned matrices). I would be glad to hear you experience with graph lasso in R or python. 
With this email, I have attached a little python code that simulates this problem in a few lines. Any input will be of great appreciation.
Thank you all,
 
Skander


#### Steps/Code to Reproduce

from sklearn.covariance import graph_lasso
from sklearn.datasets import make_spd_matrix

symetric_PD_mx= make_spd_matrix(100)

glout = graph_lasso(emp_cov=symetric_PD_mx, alpha=0.01,mode="lars")

#### Actual Results
<!-- ---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-10-9ac318eec6e7> in <module>()
----> 1 glout = graph_lasso(emp_cov=symetric_PD_mx, alpha=0.01,mode="lars")

~\AppData\Local\Continuum\anaconda3\envs\my_env\lib\site-packages\sklearn\covariance\graph_lasso_.py in graph_lasso(emp_cov, alpha, cov_init, mode, tol, enet_tol, max_iter, verbose, return_costs, eps, return_n_iter)
    222                             sub_covariance, row, Xy=row, Gram=sub_covariance,
    223                             alpha_min=alpha / (n_features - 1), copy_Gram=True,
--> 224                             eps=eps, method='lars', return_path=False)
    225                 # Update the precision matrix
    226                 precision_[idx, idx] = (

~\AppData\Local\Continuum\anaconda3\envs\my_env\lib\site-packages\sklearn\linear_model\least_angle.py in lars_path(X, y, Xy, Gram, max_iter, alpha_min, method, copy_X, eps, copy_Gram, verbose, return_path, return_n_iter, positive)
    358                 L_ = L[:n_active, :n_active].copy()
    359                 while not np.isfinite(AA):
--> 360                     L_.flat[::n_active + 1] += (2 ** i) * eps
    361                     least_squares, info = solve_cholesky(
    362                         L_, sign_active[:n_active], lower=True)

OverflowError: int too large to convert to float -->

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]
NumPy 1.14.3
SciPy 1.1.0
Scikit-Learn 0.19.1

<!-- Thanks for contributing! -->

user: SSoltani83
issue_11395: state: open
title: f_regression takes square root of negative values when constant columns are present
body: #### Description
`sklearn.feature_selection.f_regression` raises `RuntimeWarning: invalid value encountered in sqrt` when an array with any constant column is passed in. 

If this is the expected behavior, this issue should be closed. However, it seem suspicious that this only happens when `center` is `True`, and even so it may be worthwhile to check for constant columns and raise a more helpful warning.

If the call to `f_regression` below is replaced with
`f_regression(X, y, center=False)` then all values are returned with no NaN's or warnings.

I'm still not sure what conditions are needed for the error to occur. A constant column usually creates a division by zero warning, which appears to be the correct behavior. This specific array always results in square root of a negative, which is unexpected.

#### Steps/Code to Reproduce
```python
import numpy as np
from sklearn.feature_selection import f_regression

X = np.array([[  0.92  ,   0.2674,  54.934 ,  26.    ],
              [  0.92  ,   0.2674,  54.934 ,  28.    ],
              [  0.92  ,   0.2674,  54.934 ,  23.    ],
              [  0.92  ,   0.2674,  54.934 ,  21.    ],
              [  0.92  ,   0.2674,  54.934 ,  29.    ],
              [  0.92  ,   0.2674,  54.934 ,  25.    ],
              [  0.92  ,   0.2674,  54.934 ,  23.    ],
              [  0.92  ,   0.2674,  54.934 ,  28.    ],
              [  0.92  ,   0.2674,  54.934 ,  27.    ],
              [  0.92  ,   0.2674,  54.934 ,  27.    ]])
y = np.array([ 0.92,  0.92,  0.92,  0.92,  0.92,  0.92,  0.92,  0.92,  0.92,  0.92])

f_regression(X, y)
```

#### Expected Results
No warning raised and no NaN's in output
#### Actual Results

```
/usr/local/miniconda3/envs/testml/lib/python3.6/site-packages/scikit_learn-0.19.1-py3.6-macosx-10.7-x86_64.egg/sklearn/feature_selection/univariate_selection.py:292: RuntimeWarning: invalid value encountered in sqrt
  n_samples * X_means ** 2)
/usr/local/miniconda3/envs/testml/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in greater
  return (self.a < x) & (x < self.b)
/usr/local/miniconda3/envs/testml/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in less
  return (self.a < x) & (x < self.b)
/usr/local/miniconda3/envs/testml/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:1821: RuntimeWarning: invalid value encountered in less_equal
  cond2 = cond0 & (x <= self.a)
```
```python
(array([        nan, -8.        , -8.        , -8.07593079]),
 array([ nan,   1.,   1.,   1.]))
```
#### Another example without the error
```python
...
f_regression(X, y, center=False)
```
returns 
```
(array([ -2.02661983e+16,  -2.02661983e+16,  -2.02661983e+16,
          9.57231884e+02]),
 array([  1.00000000e+00,   1.00000000e+00,   1.00000000e+00,
          1.88654093e-10]))
```

#### Versions
Darwin-15.6.0-x86_64-i386-64bit
Python 3.6.5 |Anaconda, Inc.| (default, Apr 26 2018, 08:42:37) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
NumPy 1.12.1
SciPy 1.1.0
Scikit-Learn 0.19.1
user: robertmaxwilliams
issue_11379: state: open
title: OneHotEncoder doesn't handle columns with mix of string and int
body: I haven't followed the review of OneHotEncoder that closely but I figure that's a known limitation:
```python
from sklearn.preprocessing import OneHotEncoder
X2 = [['a'], ['b'], ['c'], [1]]
OneHotEncoder(categories='auto').fit_transform(X2)
```
> TypeError: '<' not supported between instances of 'int' and 'str'

I think we probably want to support this, right?
user: amueller
issue_11369: state: open
title: Proposal: Indices for Clustering.
body: I have implemented cluster validation indices  both  internal and external Indices as part of my package.
I have 40 such indices which are tested and packaged as such into a  Package called [CRAVED](https://github.com/elixir-code/craved/tree/master/craved). I would like to merge these indices as a part of SKlearns metrics for Clustering. Please let me how i can get started with this.
user: ntnshrav
issue_11368: state: open
title: [MRG] New feature: SamplingImputer
body: This is WIP to add the sampling imputation strategy as discussed in #11209

I'm not sure if we want to make it a new strategy for the ``SimpleImputer``, or a new class: ``SamplingImputer``, only dedicated to this transformation.

I made a commit for both so you can see and tell me what you think is best.
To me, just make it new strategy for the ``SimpleImputer`` is fine, since it's a simple strategy and it doesn't change the code at all (except adding the strategy).

There are still TODOs in there:

- [x] support sparse
- [x] make tests
- [x] update doc
user: jeremiedbb
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
Needs Decision - Include Feature: Requires decision regarding including feature
issue_11360: state: open
title: Add Contrastive Pessimistic Likelihood Estimation (CPLE) as semi-supervised learning method to LabelPropagation
body: #### Description
The semi-supervised LabelPropagation class offers LabelPropagation (from this 2002 paper: http://pages.cs.wisc.edu/~jerryzhu/pub/CMU-CALD-02-107.pdf) and LabelSpreading from 2004 (http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.115.3219).
Recently (2015), there has been some promising research and a new method called Contrastive Pessimistic Likelihood Estimation was introduced by Marco Loog (paper: https://arxiv.org/pdf/1503.00269.pdf). The method yields better results than maximum or simple log-likelihood estimation.

#### Suggested Change
Extend semi-supervised learning to include an implementation of CPLE.

user: plbecker
labels: Needs Decision - Close: Requires decision for closing
issue_11356: state: open
title: Support large sparse matrices in SVC/SVR
body: Support for sparse matrices having `indices`, `indptr`, `row` or `col` attributes with int64 dtype was recently added or confirmed for most sparse-supporting estimators.

LibSVM (via SVC, SVR, NuSVC, NuSVR and OneClassSVM) do not yet support such large sparse matrices (they use `accept_large_sparse=False`). We could try to fix this.
user: jnothman
issue_11355: state: open
title: Support large sparse matrices in SGD* and SequentialDataset
body: Support for sparse matrices having `indices`, `indptr`, `row` or `col` attributes with int64 dtype was recently added or confirmed for most sparse-supporting estimators.

SGDClassifier and SGDRegressor do not yet support such large sparse matrices (they use `accept_large_sparse=False`). We could try to fix this.
user: jnothman
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_11352: state: open
title: Add sample weights to calibration curve
body: As of version 0.19.1, the `calibration_curve` function does not take a `sample_weight` keyword argument to specify different weights for each value.
user: samronsin
issue_11351: state: open
title: Score Function as a metric for unsupervised clustering algorithms in Scikit-Learn
body: The **Score Function (SF)** for unsupervised clustering algorithms is not present in scikit-learn.

Score Function is described in this [paper](https://pdfs.semanticscholar.org/9701/405b0d601e169636a2541940a070087acd5b.pdf).

If this issue is a go then I can make the PR.
user: sairajk
issue_11349: state: open
title: [MRG] FIX allow non-finite target values in TransformedTargetRegressor
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #11339 
simply changed `force_all_finite` to `'False'`

**Update:**
turn off all finiteness checks
#### What does this implement/fix? Explain your changes.
Allow target values to have missing values in `TransformedTargetRegressor`


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: vahidbas
issue_11348: state: open
title: Adding support for sparse label matrix in coverage_error.
body: Currently the implementation of `sklearn.metrics.coverage_error` expects dense indicator matrix for ground-truth labels ([code](https://github.com/scikit-learn/scikit-learn/blob/a24c8b46/sklearn/metrics/ranking.py#L695-L711))

Do you think it could be beneficial to extend it to support sparse matrices for `y_true` as well?

I could open a PR, as I needed this functionality in my personal project.
user: elanmart
issue_11343: state: open
title: Only Fit is parallelized for OneVsRestClassifier
body: #### Description
`sklearn.multiclass.OneVsRestClassifier` has an `n_jobs` parameter, which runs the `fit` method in parallel for all of the sub-estimators. However, `predict` or `predict_proba` are not executed in parallel.

[Here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/multiclass.py#L343) is the code that is not parallel for `predict_proba`, and [here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/multiclass.py#L269) is the code that is parallel for `fit`.

#### Versions
master branch
user: EmreAtes
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_11339: state: open
title: Allow NaNs for the target values in TransformedTargetRegressor
body: <!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
One potential use case for  `TransformedTargetRegressor` is to get rid of missing values in the target. but currently [initial check of the fit method](https://github.com/scikit-learn/scikit-learn/blob/5718466bb5fac8fd0259127122e1107f31d79297/sklearn/compose/_target.py#L165) doesn't allow such array.

#### Steps/Code to Reproduce
Example:
```python
from sklearn.compose import TransformedTargetRegressor
from sklearn.impute import SimpleImputer
from sklearn.tree import DecisionTreeRegressor
from sklearn import datasets

X, y = datasets.load_linnerud(return_X_y=True)

## put some NaN in y
y[5, 1] = np.NaN

estimator = TransformedTargetRegressor(
    regressor=DecisionTreeRegressor(),
    func = lambda _y: SimpleImputer().fit_transform(_y), # becuse SimpleImputer doesnt have inverse
    inverse_func = lambda _y: _y,
    check_inverse = False
)

estimator.fit(X, y)
```

This raises:
`ValueError: Input contains NaN, infinity or a value too large for dtype('float64').`

user: vahidbas
issue_11331: state: open
title: Make sklearn.feature_selection.f_classif raise warn on float y?
body: Is there a way to have `f_classif` raise a warning if the `y` vector contains a floating point number? This would be useful if someone forgets to change the default `score_func` arg to, say, `sklearn.feature_selection.SelectPercentile`. Example:
```
Input:
import numpy as np
from sklearn.feature_selection import f_classif

X = np.random.random((10,3))
y = 3*X[:,0] + 7*X[:,1] + 11*X[:,2]
f_classif(X,y)

Output:
/usr/local/lib/python3.6/site-packages/sklearn/feature_selection/univariate_selection.py:109: RuntimeWarning: invalid value encountered in true_divide
  msw = sswn / float(dfwn)
(array([nan, nan, nan]), array([nan, nan, nan]))
```
user: qpwo
issue_11324: state: open
title: [WIP] API specify test parameters via classmethod
body: This is an initial implementation of what I suggested in https://github.com/scikit-learn/scikit-learn/pull/8022#discussion_r196659219 (ping @amueller).

The idea is to put the test configuration for an estimator class on the class. This provides:
* advantage: can instantiate meta-estimators which have required args
* advantage: can see the test parametrisation clearly on the class
* disadvantage: code limiting `max_iter`, etc., is repetitive (this PR adds lines and decentralises functionality)
* disadvantage: harder for reviewers to point out to a contributor that these changes are needed

The key changes are to [`base.py`](https://github.com/scikit-learn/scikit-learn/pull/11324/files#diff-f392a01b355b04b8c90c84175d118fdc), [`test_common.py`](https://github.com/scikit-learn/scikit-learn/pull/11324/files#diff-b6e2c29089c86706dc868c7bfd58c1d6) and [`estimator_checks.py`](https://github.com/scikit-learn/scikit-learn/pull/11324/files#diff-a95fe0e40350c536a5e303e87ac979c4)

TODO:
* [ ] documentation
* [ ] use `_generate_test_params` when calling check_estimator on a class
* [ ] replace `check_parameters_default_constructible`
* [x] work out how to still do `check_no_attributes_set_in_init`
* [ ] test exception when running check_estimator on class requiring parameters
user: jnothman
issue_11316: state: open
title: Refactor tests for sample weights
body: In various parts of the code, we have tests for `sample_weight` support, including in metrics, and for individual estimators. we have some common [estimator checks](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/estimator_checks.py) for `class_weight`, but not really for `sample_weight` functionality (only for weight type invariance).

Recent implementations of `sample_weight` include #10933 (KMeans) and #10803 (density estimation). But as well as estimators we have things like [common tests for evaluation metrics](https://github.com/scikit-learn/scikit-learn/blob/83d9a40953d26251ec1f3e0794d6733a82f36d9f/sklearn/metrics/tests/test_common.py#L981).

Invariance testing for sample weights should include:
* `sample_weight=np.ones(len(X))` makes the same model as `sample_weight=None`
* `sample_weight=random` can make a different model to `sample_weight=None`
* `sample_weight=s` for integer array `s` makes the same model as `X=np.repeat(X, s, axis=0), y=np.repeat(y, s, axis=0)` (although there may be exceptions to this depending on how the estimator defines iteration, convergence, etc., as in #11236)
* `sample_weight=s * k` for array `s` and positive constant `k` makes the same model as `sample_weight=s`

I wonder if it is possible to establish a generic test for this, e.g. something like:

```py
def check_sample_weight_invariance(data_args, fit, is_equal):
    """
    Parameters
    ----------
    data_args : dict
        Keyword arguments to pass to fit, and which would need to be repeated
        to test equivalence to integer sample weights.
    fit : callable
        Passed data args, returns a model that can be compared with is_equal
    is_equal : callable
        Passed two models returned from fit, returns a bool to indicate equality
        between models
    """
```
user: jnothman
labels: Moderate: Anything that requires some knowledge of conventions and best practices
module:test-suite: everything related to our tests
issue_11312: state: open
title: Improve efficiency of helper functions in metrics
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
According to the line-profiling results, the `_check_targets` and `unique_labels ` takes up the majority of running time in calculating evaluation metrics. As can be seen in #11179 and #8126.

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->
https://github.com/ShangwuYao/support-issues/blob/master/testclassification.py

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
`_check_targets` and `unique_labels` should be a lot faster, or avoid unnecessarily repeated call of these functions.

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
Line-profiling on `precision_recall_fscore_support` shows that for this example, the running time is 0.026829 s, and the most time-consuming part are (sum up to 97.5% of running time).
```
Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   177         1       4546.0   4546.0     16.9      y_type, y_true, y_pred = _check_targets(y_true, y_pred)
   179         1      11116.0  11116.0     41.4      present_labels = unique_labels(y_true, y_pred)
   244         1       5232.0   5232.0     19.5          y_true = le.transform(y_true)
   245         1       5292.0   5292.0     19.7          y_pred = le.transform(y_pred)
```

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
Darwin-16.7.0-x86_64-i386-64bit
Python 3.6.1 |Anaconda custom (64-bit)| (default, May 11 2017, 13:04:09)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
NumPy 1.12.1
SciPy 1.1.0
Scikit-Learn 0.20.dev0

<!-- Thanks for contributing! -->

user: ShangwuYao
issue_11305: state: open
title: [MRG] Pandas Interoperability section
body: This PR aims to implement bits of #10453 as part of the python-sprints London doc sprint.

More details such as which bits will be worked on and by whom will follow as part of the sprint.
user: fdroessler
issue_11304: state: open
title: Dunn Index as a metric for unsupervised clustering algorithms 
body: The **Dunn Index(DI)** for unsupervised clustering algorithms is not present in sklearn.

It is described [here](https://en.wikipedia.org/wiki/Dunn_index) . 

If this issue is a go then I can make the PR
user: 53X
issue_11300: state: open
title: [RFC] Deprecate n_samples_seen_ in MaxAbsScaler and MinMaxScaler
body: `MinMaxScaler` and `MaxAbsScaler` have an attribute `n_samples_seen` which is not used to compute the statistics. For instance this is useful to have it in `StandardScaler` but does not bring anything in those 2 scalers.

In #11011, @jnothman proposed to deprecate this parameter which I am also in favor of.

@amueller @ogrisel Any opinion and historical reasons why this attribute is there (because the class have a ``partial_fit`` method?
user: glemaitre
labels: Needs Decision: Requires decision
issue_11282: state: open
title: Fix for #10372: partial_dependence.py _grid_from_X ignores percentile specification for small numbers of unique feature values
body: #### Reference Issues/PRs
Fixes #10372

#### What does this implement/fix? Explain your changes.
We should accept the recommended idea.

When there are a small number of unique values along a particular dimension, we could build a smaller grid by just picking those unique values.
This PR can make this grid slightly more efficient by just considering the unique values inside the specified percentile range.
user: haroldfox
labels: Needs Decision: Requires decision
issue_11267: state: open
title: Feature Request: DictVectorizer with partial_fit()
body: #### Description
While training a classifier with partial_fit I'm using FeatureHasher to deal with different features in batches, the problem is that I need the feature_names mapping.

Does it make sense to add a `n_features` parameter and a `partial_fit` method to `DictVectorizer`? The idea is that the output matrix would always have the same dimension and new feature_names would be added to `vocab `on the fly.

#### How it would work
Add a new `n_features` parameter (with `default=None`). If `n_parameters` is different than `None` we would add dummy features to `vocab`.
```
v = DictVectorizer(sparse=False, n_features=5)
D = [{'foo': 1, 'bar': 2}, {'foo': 3, 'baz': 4}]
X = v.partial_fit(D)
X
```
array([[2., 0., 1., 0., 0.],
       [0., 4., 3., 0., 0.]])

```
v.vocabulary_
```
{'3': None, '4': None, 'bar': 0, 'baz': 1, 'foo': 2}

```
D = [{'foo': 1, 'bar': 2, 'baz': 4, 'new_feature': 9}]
X = v.partial_fit(D)
X
```
array([[2., 4., 1., 9., 0.]])
```
v.vocabulary_
```
{'4': None, 'bar': 0, 'baz': 1, 'foo': 2, 'new_feature': 3}

Does it make sense to have a `Transformer` with a `partial_fit()` method?

Thanks!
user: dmesquita
issue_11266: state: open
title: [WIP] API: allow cross validation to work with partial_fit
body: #### Reference Issues/PRs
Addresses (but doesn't fix) #1626: "API Proposal: Genearlized Cross-Validation and Early Stopping". This PR only address early stopping on models that have `partial_fit` (https://github.com/scikit-learn/scikit-learn/issues/1626#issuecomment-12739541), not addressing "when more computation doesn't improve the result." (https://github.com/scikit-learn/scikit-learn/issues/1626#issue-10319901).

I will use this in https://github.com/dask/dask-searchcv/pull/72. Briefly, I want to repeatedly use `cross_validate` with a specified number of `partial_fit` calls.

#### What does this implement/fix? Explain your changes.
This PR implements early stopping for cross validation when `partial_fit` is present. This requires that

- `_fit_and_score` have a `partial_fit` keyword argument that can either be a bool or an int that reflects the number of times `est.partial_fit` is called.
  - I also add a `partial_fit` keyword arg to `cross_validate`, which relies on `_fit_and_score`.
- the specification of a different train/validation set if `_fit_and_score` is called repeatedly.
  - yes, it's possible to make sure train/validation are always separated on repeated calls but not really user-friendly and doesn't have the cleanest documentation.

#### TODO
-  ~~make work with pipelines~~
- [x] allow for specification of a separate train and validation set as mentioned in https://github.com/scikit-learn/scikit-learn/issues/1626#issue-10319901
- [x] make sure will as expected when called repeatedly, which comes up in https://github.com/dask/dask-searchcv/pull/72.
-  ~~integrate with GridSearchCV and RandomizedSearchCV~~
user: stsievert
labels: Needs Decision: Requires decision
issue_11263: state: open
title: Different results depending on data type of class labels using svm.SVC
body: Hi All, 

I'm getting different results using sklearn.svm.SVC depending on whether I use strings or integers for my class labels during multiclassification. I would expect that the data type for class labels shouldn't matter (unless I am missing something). I've narrowed this down to SVC (i.e. this doesn't happen with LogisticRegression() or LinearSVC()). I've also _not_ been able to reproduce this issue with a toy dataset made from scratch, but below I've provided a sample (n = 20) of the dataset I am working with, along with code, that produces this issue. 

Link to the data:
https://www.dropbox.com/s/tgx02j5nfvc860q/test.csv?dl=0

The first 5 rows and 7 columns look like this (columns 0 and 1 are class labels, either as string or int, respectively):

```    0  1         2         3         4         5         6         7  
0  LL  0  0.728522 -0.649361  3.853140 -0.903645 -1.551013 -0.080372   
1  LL  0  1.601109 -4.050748 -0.429045 -3.481833  2.344579 -0.615784   
2  LL  0  3.829118  1.465271  3.090473 -4.780373  1.583611  3.498092   
3  LL  0 -0.535664  1.090892  3.537763 -1.333069  0.273352  2.050236   
4  LL  0 -0.413857  0.795902  1.273712  0.621104  2.638997  2.792364
```

#### Steps/Code to Reproduce
```import numpy as np
import pandas as pd
from sklearn.svm import SVC
from sklearn.cross_validation import cross_val_score

data = pd.read_csv('test.csv')

x = data.iloc[:, 2:].values # features with shape (20, 44)
class_str = data.iloc[:, 0]  # contains class labels 'LL', 'LR', 'GL', 'GR'
class_int = data.iloc[:, 1] # contains class labels 0 1 2 3 

# run a simple 3-fold CV on the data with each type of class label
clf = SVC(C=1.0, kernel='linear')
print(cross_val_score(clf, x, class_int))
print(cross_val_score(clf, x, class_str))
```

#### Expected Results
Same cross validation accuracies. E.g., 

```
[0.625 0.25  0.25 ]
[0.625 0.25  0.25 ]
```

<!-- Example: No error is thrown. Please paste or describe the expected results.-->

#### Actual Results

Different cross validation accuracies:
```
[0.625 0.25  0.25 ]
[0.625 0.375 0.5  ]
```
So, the results differ based on the label type. But, critically, the int and str labels properly correspond to one another so we wouldn't expect there to be a mismatch.

Any help would be appreciated, thanks!

#### Versions
```
Darwin-17.4.0-x86_64-i386-64bit
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:14:23) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
NumPy 1.14.2
SciPy 1.0.1
Scikit-Learn 0.19.1
```

<!-- Thanks for contributing! -->
user: danjgale
issue_11209: state: open
title: Add strategy="random" to SimpleImputer
body: The purpose would be to provide an unbiased, stochastic, univariate imputer. The imputer would replace missing values by values sampled uniformly at random from the non-missing values of the sample column.

This would require to add additional `random_state` constructor parameter to the class.

As for #11208 this strategy should work both on numerical and non-numerical dtypes.

user: ogrisel
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_11207: state: open
title: Obtain model loss for arbitrary data
body: 
#### Description
Feature Request: Obtain model loss for arbitrary data.

#### Steps/Code to Reproduce
Whenever an `sklearn` model is fit to some data, it minimizes some loss function.  It would be useful to obtain the model loss on arbitrary data using that loss function.  Note that this is a common operation in other ML frameworks.

#### Expected Results
```
model = sklearn.linear_model.LogisticRegression().fit(X_train,y_train)
model.get_loss(X_train, y_train) #gives the loss for these values
model.get_loss(X_test, y_test) #gives the loss for other values
```

#### Actual Results
None


user: csbrown
issue_11203: state: open
title: FunctionTransformer applied to each row/col?
body: In using FunctionTransformer, particularly for things like feature extraction (e.g. as a pipeline step before DictVectorizer), I've made the mistake of thinking that the function would be applied to each row of `X` rather than all of `X` at once.

Would it help to have a parameter such as:

```
    over : {'all', 'row', 'col', 'cell'}
		If 'all' (default) the whole of X is passed to the function for
		transformation.  If 'row', each element of X (e.g. each row) is passed
		to the function, and result is composed as a list (array???) of
		function return values. 'col' is like 'row' but for transposed X and
		transposed result.
```
user: jnothman
issue_11181: state: open
title: Deprecate with_mean in StandardScaler?
body: I think we should deprecate ``with_mean`` in ``StandardScaler`` since it leads to weird behavior. For sparse data, users should use ``MaxAbsScaler``.
user: amueller
labels: Needs Decision: Requires decision
issue_11178: state: open
title: SGDRegressor gets poor fit with sparse matrix
body: If a sparse matrix is passed into the function, it doesn't throw any error but gets a poor fit **sometimes**. I think it should either throw an error when the parameter is a sparse matrix or convert it into a dense matrix like https://github.com/scikit-learn/scikit-learn/pull/535

Below is some code to compare the differences. (It fits well on boston housing price even when the parameter is a sparse matrix but fails on diabetes.)

```python
import numpy
from scipy.sparse import csr_matrix
from sklearn.datasets import load_diabetes
from sklearn.linear_model import SGDRegressor, LinearRegression
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

diabetes = load_diabetes()
X = diabetes.data
# X = csr_matrix(X)  # Uncomment this line to use a sparse matrix
y = numpy.asarray(diabetes.target)

scaler = StandardScaler(with_mean=False)
scaler.fit(X)
X = scaler.transform(X)

estimator = SGDRegressor()
# estimator = LinearRegression()  # LinearRegression works with sparse matrix
estimator.fit(X, y)
predicted = estimator.predict(X)

fig, ax = plt.subplots()
ax.scatter(y, predicted, edgecolors=(0, 0, 0))
ax.plot([y.min(), y.max()], [y.min(), y.max()], 'k--', lw=4)
ax.set_xlabel('Measured')
ax.set_ylabel('Predicted')
plt.show()
```
Just stating in the documentation that the input should be a dense matrix is not enough, it may cause surprises like this.

NumPy 1.14.2
SciPy 1.1.0
Scikit-Learn 0.19.1

user: jennaliu
issue_11175: state: open
title: ENH: add oob support to CalibratedClassifier
body: ## Enhances: CalibratedClassifierCV
This pull request implements out-of-bag (oob) support for `CalibratedClassifierCV` when using a base_classifier that uses out-of-bag (e.g. `RandomForestClassifier`).

If the base_classifier has the attribute `oob_decision_function_` after training (which means that the base classifier was trained with `oob=True`) the calibrated classifier uses the fitted calibration to transform the out-of-bag predictions using that fit. 

Importantly, since the oob prediction is calculated for each fold, the model holds a matrix of the oob predictions for all samples in each fold and embeds the predictions for the fold (leaving nan values for the left-out fold). After the cross-validation the matrix is averaged (ignoring nan values) to obtain an oob prediction for all the samples.

Finally, the `oob` attribute is set to True

###note:
I could not find a specific issue that this resolves.
user: yishaishimoni
labels: Needs Decision: Requires decision
issue_11174: state: open
title: [WIP] add max_iter support for partial_fit in SGD and PassiveAgressiveClassifier
body: This PR adds control over how many examples `partial_fit` sees. That is, how
many epochs (/passes over the training set) should `partial_fit` observe before
returning? I tried to add these to all classifiers listed in [User guide:
Scaling strategies][1], but some of them already had support for this.

This PR adds a `max_iter` keyword arg to `PassiveAgressiveClassifier.partial_fit`
and `BaseSGDClassifier.partial_fit`.

This change amounts to moving a hard coded constraint to a user-facing keyword arg.

TODO: test.

[1]:http://scikit-learn.org/stable/modules/scaling_strategies.html

user: stsievert
labels: Needs Decision: Requires decision
issue_11165: state: open
title: Tree calculation - comparing double to float
body: #### Description
When training RandomForestClassifier using 10 trees on wine dataset.
The path of the last tree gets to the wrong leaf.

The reason is when comparing the threshold to the actual value in _file: _tree.pyx at line 883_ (or 812) we compare float (feature value) to double (threshold):

```py
if feature_value <= node.threshold:
    node = &self.nodes[node.left_child]
else:
    node = &self.nodes[node.right_child]
```

C is taking the first type, so the actual code here is:

```py
if feature_value <= (float) node.threshold:
    node = &self.nodes[node.left_child]
else:
    node = &self.nodes[node.right_child]
```
  

#### Steps/Code to Reproduce
```py
from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
wine = datasets.load_wine()
X = wine.data
split_idx = int(X.shape[0] * 0.4)
X_train = X[:split_idx]
y_train = wine.target[:split_idx]
X_validate = X[split_idx:][81].reshape(1, -1)
model = RandomForestClassifier(n_estimators=10, random_state=0)
model.fit(X_train, y_train)
mat = model.decision_path(X_validate)
print("feature 1 value is:", X_validate[0][1])
print("Visiting node number 4 in the 10th tree:", mat[0][0, 55])
print("Visiting node number 5 in the 10th tree:", mat[0][0, 56])
```

#### Expected Results
The last tree starts at node 51, the third node (number 54) is:
    feature number: 1
    with threshold: 1.89999998
    right child: 5
    left child: 4

As seen in the code the value in this feature is 1.9 --> feature_value > node.threshold --> should go right
But as shown in the code snippet, we visit node number 4 and not visiting number 5

#### Actual Results
On the 3'rd split in the 10th tree going to the wrong direction

#### Versions
Linux-4.13.0-37-generic-x86_64-with-Ubuntu-16.04-xenial
Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609]
NumPy 1.14.2
SciPy 1.0.0
Scikit-Learn 0.19.1


#### Possible Solution
Kind of a hacky solution, but I think it will work.
Switch the order in the if statement, and we will get:
```py
if node.threshold >= (double) feature_value :
    node = &self.nodes[node.left_child]
else:
    node = &self.nodes[node.right_child]
```
Because double is more precise than float, we don't loss information here. 
Not sure if and how it will affect the training

user: yuvaljacoby
issue_11117: state: open
title: Common test to check that __init__ doesn't mess with parameters
body: There's ``check_dont_overwrite_parameters`` which checks that ``fit`` doesn't overwrite ``__init__`` params, and there's ``check_no_attributes_set_in_init`` that checks that only ``__init__`` attributes are set. There is not check that they are actually set to the correct value, so
```python
class DumbOne(BaseEstimator):
    def __init__(param=0):
        param = 1
```
actually passes the common tests. That's not good imho.
user: amueller
labels: Needs Decision: Requires decision
module:test-suite: everything related to our tests
issue_11096: state: open
title: [MRG] Implement calibration loss metrics
body: See discussion on issue #10883.
Closes #18268.

This PR implements calibration losses for binary classifiers.

It also updates the doc about calibration, especially inaccurate references to the Brier score.
user: samronsin
issue_11086: state: open
title: Quantile Regression Forest [Feature request]
body: Hi, Scikit-learn owners and contributors,

I am wondering whether scikit-learn want to implement the quantile regression forest: http://www.jmlr.org/papers/volume7/meinshausen06a/meinshausen06a.pdf. 

It is a quite highly cited paper, and already has an R package. https://cran.r-project.org/web/packages/quantregForest/quantregForest.pdf

This method has been widely used in various quantile regression problems.

From implementation perspective, it is also a natural extension of random forest, given scikit-learn already has a good random forest implementation. Most of the computation is performed with random forest base method.

In addition, R's extra-tree package also has quantile regression functionality, which is implemented very similarly as quantile regression forest. So if scikit-learn could implement quantile regression forest, it would be an relatively easy task to add it to extra-tree algorithm as well. 

Please let me know if it is possible, Thanks.


user: RLstat
issue_11071: state: open
title: Importing sklearn messes with CPU affinity, causing n_jobs=-1 to become useless
body: #### Description
Please see [a PoC notebook](https://nbviewer.jupyter.org/gist/Pastafarianist/471be9b46ab4fa5518ae74724a644fe7?flush_cache=true).

**tl:dr:** `import sklearn` causes CPU affinity to be set to `{0}`. Resetting it causes `RandomizedSearchCV` to freeze.

#### Expected Results
`n_jobs=-1` uses all cores and does not freeze.

#### Actual Results
Only one core is loaded at 100% and nothing happens for 30+ minutes. Not resetting CPU affinity or not specifying `n_jobs=-1` causes the freeze to go away.

#### Versions

```
Linux-4.14.36-1-MANJARO-x86_64-with-arch-Manjaro-Linux
Python 3.6.5 (default, Apr 12 2018, 22:45:43) 
[GCC 7.3.1 20180312]
NumPy 1.14.2
SciPy 1.0.1
Scikit-Learn 0.19.1
```
user: dniku
issue_11054: state: open
title: [MRG] Added stratify option for learning_curve
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #10684

#### What does this implement/fix? Explain your changes.
Added support of stratified splits of training data to learning_curve method by using StratifiedShuffleSplit.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: astrakhantsev
issue_11046: state: open
title: [MRG] Mean shift: density based cluster assignment
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Background
The Mean Shift algorithm in scikit-learn is not implemented as described in the reference cited in the docs (Comaniciu and Meer 2002). The issue is the labelling of the data points, i.e. cluster assignment: in scikit-learn data points are assigned to the nearest centroid. It is evident from Fig 2. in (Comaniciu and Meer 2002) that this is not what is done there. Moreover, the paper states that

”The delineation of the clusters is a natural outcome of the mode seeking process. After convergence, the basin of attraction of a mode, i.e., the data points visited by all the mean shift procedures converging to that mode, automatically delineates a cluster of arbitrary shape.” [p.  608, Section 3]

This implies that a data point should be assigned to its attractor in the mode-seeking process, i.e. to the point it converges to.

+ Dorin Comaniciu and Peter Meer, “Mean Shift: A robust approach toward feature space analysis”. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2002. pp. 603-619

#### What does this implement/fix? Explain your changes.
I have added an option `cluster_assignment` to the MeanShift and mean_shift functions that can be set to `nearest_centroid` (default) to get the current scikit-learn cluster assignment and to `attractor` to get assignment as described above. When bin seeding is used together with `attractor` cluster assignment, a data point is assigned to the same mode as the seed closest to it.

The advantage of using `attractor` cluster assignment is that the cluster shapes adapts to the density of the data. I have chosen a dataset for the example in examples/cluster/plot_mean_shift.py that showcases how the clustering can be different with the two `cluster_assignment` options. When an outlier is chosen as cluster center, with `nearest_centroid` also other points can be assigned to this cluster, something that is avoided with the `attractor` option. An outlier does not attract any data points in the mode-seeking process and will be the single data point in the cluster.

#### Any other comments?
I have noted that with coarse bin seeding and the `attractor` option, the clusters shapes will be awkward - a collection of rectangles pasted together. To remedy this I put the default bin size for the `attractor` option to 0.2 times the bandwidth; the default for `nearest_centroid` equals the bandwidth as before.  This affects the complexity of the algorithm and a paragraph addressing this has been added to the documentation of the MeanShift class, under Notes/Scalability:
```
With cluster_assignment = 'nearest_centroid', the default bin size equals
the bandwidth, whereas for cluster_assignment = 'nearest_centroid' the
default bin size is 0.2 times the bandwidth. This setting gives
'attractor' approximately 5^n_features times as many seeds.
```



<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->
user: kjohnsson
labels: Needs Decision: Requires decision
issue_11020: state: open
title: SVC's max_iter default setting of -1 can cause very long running times
body: The class `sklearn.svm.SVC` has parameter `max_iter=-1` by default. This causes the optimizer to have no maximum number of iterations, and can cause the classifier to run very long, when solving hard problems. Unfortunately I have no steps to reproduce the long running behaviour, as this happened with my own, real world data. 

```python
from sklearn.svm import SVC

svc = SVC()
print(svc.get_params()['max_iter'])
```

As a change I would recommend to set `max_iter=1000` by default. This is also the default in `sklearn.svm.LinearSVC`. People can then decide themselves if they want to run the solver for longer, if they think that is worth it.
user: ghost
labels: Needs Decision: Requires decision
issue_11000: state: open
title: Preserving dtype for float32 / float64 in transformers
body: This is the issue which we want to tackle during the Man AHL Hackathon.

We would like that the transformer does not convert float32 to float64 whenever possible.  The transformers which are currently failing are:

- [x] BernoulliRBM (#24318)
- [x] Birch (#22968)
- [x] CCA - inherits from PLS, so also not applicable
- [x] DictionaryLearning (#22002)
- [ ] FactorAnalysis (#13303)
- [x] FastICA
- [ ] FeatureAgglomeration (#24346)
- [x] GaussianRandomProjection (#22114)
- [x] GenericUnivariateSelect
- [x] Isomap (#24714)
- [x] LatentDirichletAllocation (#22113)
- [x] LinearDiscriminantAnalysis (#13273)
- [ ] LocallyLinearEmbedding (#24337)
- [x] LogisticRegression (#13243)
- [x] MiniBatchDictionaryLearning
- [x] MiniBatchSparsePCA (#22111)
- [x] NMF
- [x] PLSCanonical   - not applicable as both `X` and `y` are used
- [x] PLSRegression  - not applicable as both `X` and `y` are used
- [x] PLSSVD   -  not applicable as both `X` and `y` are used
- [x] RBFSampler (#24317)
- [x] RidgeRegresssion (#13302)
- [x] SGDClassifier/SGDRegressor (#9084)(#13346)
- [x] SkewedChi2Sampler
- [x] SparsePCA (#22111)
- [x] SparseRandomProjection (#22114)

We could think to extend it to integer whenever possible and applicable.

Also the following transformers are not included in the common tests. We should write a specific test:

```python
# some strange ones                                                                                                                                                         
DONT_TEST = ['SparseCoder', 'DictVectorizer',                                                                                                                               
             'TfidfTransformer',                                                                                                                     
             'TfidfVectorizer' (check 10443), 'IsotonicRegression',                                                                                                                       
             'CategoricalEncoder',                                                                                                 
             'FeatureHasher',                                                                                             
             'TruncatedSVD', 'PolynomialFeatures',                                                                                                                          
             'GaussianRandomProjectionHash', 'HashingVectorizer',                                                                                                           
             'CountVectorizer']
```

We could also check classifiers, regressors or clusterers (see https://github.com/scikit-learn/scikit-learn/issues/8769 for more context),

- [ ] AffinitiyPropagation -> Bug in #10832
- [ ] check SVC -> #10713 


Below the code executed to find the failure.
<details>

```python
# Let's check the 32 - 64 bits type conservation.                                                                                                                       
if isinstance(X, np.ndarray):                                                                                                                                           
    for dtype in [np.float32, np.float64]:                                                                                                                              
        X_cast = X.astype(dtype)                                                                                                                                        
                                                                                                                                                                            
        transformer = clone(transformer_orig)                                                                                                                           
        set_random_state(transformer)                                                                                                                                   
                                                                                                                                                                            
        if hasattr(transformer, 'fit_transform'):                                                                                                                       
            X_trans = transformer.fit_transform(X_cast, y_)                                                                                                             
        elif hasattr(transformer, 'fit_transform'):                                                                                                                     
            transformer.fit(X_cast, y_)                                                                                                                                 
            X_trans = transformer.transform(X_cast)                                                                                                                     
                                                                                                                                                                            
        # FIXME: should we check that the dtype of some attributes are the                                                                                              
        # same than dtype.                                                                                                                                              
        assert X_trans.dtype == X_cast.dtype, 'transform dtype: {} - original dtype: {}'.format(X_trans.dtype, X_cast.dtype)
```
</details>



## Tips to run the test for a specific transformer:

- Choose a transformer, for instance `FastICA`
- If this class does not already have a method named `_more_tags`: add the following code snippet at the bottom of the class definition:

```python
    def _more_tags(self):
        return {"preserves_dtype": [np.float64, np.float32]}
```
- Run the common tests for this specific class:
```
pytest sklearn/tests/test_common.py -k "FastICA and check_transformer_preserve_dtypes" -v
```
- It should fail: read the error message and try to understand why the `fit_transform` method (if it exists) or the `transform` method returns a `float64` data array when it is passed a `float32` input array.

It might be helpful to use a debugger, for instance by adding the line:

```python
import pdb; pdb.set_trace()
```

at the beginning of the `fit_transform` method and then re-rerunning pytest with:

```
pytest sklearn/tests/test_common.py -k "FastICA and check_transformer_preserve_dtypes" --pdb
```

Then using the `l` (list), `n` (next), `s` (step into a function call), `p some_array_variable.dtype` (`p` stands for print) and `c` (continue) commands to interactively debug the execution of this `fit_transform` call.


ping @rth feel free to edit this thread.
user: glemaitre
labels: Hard: Hard level of difficulty
Meta-issue: General issue associated to an identified list of tasks
float32: Issues related to support for 32bit data
issue_10976: state: open
title: learning_curve: Separate scores for multioutput estimators
body: #### Description
Many linear estimators can take several target vectors at once, like `Ridge`. The `learning_curve` function nicely supports this by allowing the value of `y` to be an array of any shape. But then, I would like to plot the learning curve for each target vector independently. Which the function doesn't allow me to do. Down the road, the scorer just squash everything together and produce a single number. Which prevent me from getting one score per target vector.

I might try to side-step the issue by providing a multimetric scorer, but the `learning_curve` function doesn't currently support that. I guess supporting multimetric scorers would be the easiest way to support separate scores for multioutput estimators.

#### Steps/Code to Reproduce
```python
#!/usr/bin/env python3

import numpy as np
from sklearn import model_selection
from sklearn import linear_model

def main():
    X = np.random.random((100, 1))
    y = np.random.random((100, 2))

    reg = linear_model.Ridge(1e-6)
    train_sizes, tscores, vscores = model_selection.learning_curve(reg, X, y,
            train_sizes=[66], scoring='neg_mean_squared_error')

    print(tscores)
    print(vscores)

if __name__ == '__main__':
    main()
```

#### Expected Results
Return one score per target vector.

#### Actual Results
Return a global metric for all the target vectors.

#### Versions
```
>>> import platform; print(platform.platform())
Linux-4.15.0-2-amd64-x86_64-with-debian-buster-sid
>>> import sys; print("Python", sys.version)
Python 3.6.5rc1 (default, Mar 14 2018, 06:54:23) 
[GCC 7.3.0]
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.14.0
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 0.19.1
>>> import sklearn; print("Scikit-Learn", sklearn.__version__)
Scikit-Learn 0.19.1
```

user: Celelibi
issue_10973: state: open
title: Feature discussion: callbacks for long operations
body: Ideally, I think, operations that take a lot of time should both:
- have a way of showing progress, and
- be interruptable.

It was previously discussed at least in #78, #7574, #7596. 

Regarding progress, in #7574 @amueller proposed not bake progress bar in, but rather add callbacks. In #7596 @denis-bz suggested to have callbacks that are passed locals(), which is an interesting idea. I also saw that `fit()` in  `GradientBoostingClassifier` has the `monitor` parameter, which makes showing progress bars easy.

In #7596, @denis-bz suggested callbacks can also be used to interrupt computation. In our project, Orange (https://github.com/biolab/orange3), we do something similar: sometimes, where there is no other available mechanism, we raise a BaseException inside a callback to interrupt running threads.

Orange uses scikit-learn a lot and lack of callbacks in scikit-learn makes showing progress or interrupting hard (we'd like to allow stopping of running computations). For now, we have to resort to hacks. For example, inour Neural Network widget, we subclassed scikit-learn NNs and added a callback on `n_iter_` change (biolab/orange3#2958)

We would like to help in implementing callbacks, but first, we are asking if you would even consider having something similar to  `GradientBoostingClassifier` `monitor` in the other classes. What do you think?

Then, we could try thinking of an interface together and slowly start adding it to certain classed.
user: markotoplak
issue_10955: state: open
title: Feature Request: cost curve with Scikit-Learn
body: Hi everybody,

I would like to know how we can draw a cost curve with Scikit-Learn.
Cost curves analyse classification performance by varying operating points. Operating points are based on class probabilities and misclassification costs (see p5 of http://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0118432&type=printable).

Thanks
user: AlexandraBomane
issue_10944: state: open
title: W and H switched in NMF docs?
body: The NMF docstring says
the objective is
```
0.5 * ||X - WH||_Fro^2
````
but X is ``(n_samples, n_features)`` for us and ``W`` are the coefficients (**W**eights) and H is the latent (**H**idden) representation.
So this should be HW (in the literature it's often WH because their X is transposed).

And have the following example:
```
    >>> W = model.fit_transform(X)
    >>> H = model.components_
```
The example at least is wrong/confusing. H should be the transformed data.

I assume the example is wrong because someone got confused by the objective?
I'm not sure if it's wrong in other places.


EDIT: link to the original paper:

http://www.cs.columbia.edu/~blei/fogm/2020F/readings/LeeSeung1999.pdf

![image](https://user-images.githubusercontent.com/89061/154265246-0b818066-5968-4868-847f-9fff824ea8e8.png)

user: amueller
labels: Needs Decision: Requires decision
issue_10942: state: open
title: Feature Request: Get intermediate positions for t-SNE for animation
body: Hello!

I worked with your t-SNE method for quite a while but always was mesmerized to look at animations thereof, which is not provided by your library. The probably best approach to this would be to allow a user to retrieve the intermediate steps and not just the final positions. Any interest in a pull-request to change that?

Even O'Reilly is monkey-patching this into your gradient descent method to get it running:
https://github.com/oreillymedia/t-SNE-tutorial#algorithm
user: benelot
issue_10924: state: open
title: Simplify Elkan k-means?
body: I found this cool paper that says that *removing* some of the tests in elkans algorithm can speed it up:
http://proceedings.mlr.press/v48/newling16.pdf

The paper also looks at a simplified ying-yang algorithm, which I still think is interesting (apparently mostly for low-dim spaces and when Elkans needs too much RAM).
user: amueller
issue_10912: state: open
title: MultiOutputRegressor: Support for matrices of sample_weight
body: #### Description
With `MultiOutputRegressor.fit`, It would be nice to be able to provide one `sample_weight` vector per output. The gain would just be the ease of use when you have several regression to perform at once instead of having necessarily one estimator per regression. This shouldn't be difficult to implement.

I think it would be nice also to have this directly supported by the underlying estimators that support multiple outputs (like most linear regressors). However, I have no idea how hard it could be to implement.

#### Steps/Code to Reproduce
Expected usage example.
```python
#!/usr/bin/env python3

import numpy as np
import sklearn as sk
import sklearn.linear_model
import sklearn.multioutput

X = np.random.random((10, 10))
y = np.random.random((10, 2))
w = np.random.random((10, 2))
reg = sk.linear_model.LinearRegression()
reg = sk.multioutput.MultiOutputRegressor(reg)
reg.fit(X, y, w)
```

#### Expected Results
Perform the multiple regressions using one column of `w` as `sample_weight` per call to fit to the underlying estimator.

#### Actual Results
Unsupported yet.

#### Versions
```
Python 3.6.5rc1 (default, Mar 14 2018, 06:54:23) 
[GCC 7.3.0]
NumPy 1.14.0
SciPy 0.19.1
Scikit-Learn 0.19.1
```

user: Celelibi
labels: Needs Decision: Requires decision
issue_10910: state: open
title: Bug function: reconstruct_from_patches_2d
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
Function `reconstruct_from_patches_2d` is getting the right answer!!!

#### Steps/Code to Reproduce

Code:
```python
from time import time
import numpy as np
from sklearn.feature_extraction.image import extract_patches_2d
from sklearn.feature_extraction.image import reconstruct_from_patches_2d

img = np.ones((128, 256))

height, width = img.shape


# Extract all reference patches from the left half of the image
print('Extracting reference patches...')
t0 = time()
patch_size = (128, 128)
data = extract_patches_2d(img, patch_size)
#data = data.reshape(data.shape[0], -1)
#intercept = np.mean(data, axis=0)
#data -= intercept
print('done in %.2fs.' % (time() - t0))

b = reconstruct_from_patches_2d(data, img.shape)
```


#### Actual Results
b should be all ones!!!, The result is NOT!!!

#### Versions

Windows-10-10.0.16299-SP0
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]
NumPy 1.14.2
SciPy 1.0.1
Scikit-Learn 0.19.1


<!-- Thanks for contributing! -->

user: jiadongdan
issue_10882: state: open
title: OneVsRestClassifier's fit method does not accept kwargs
body: #### Description
The `fit` method of `OneVsRestClassifier` and other MetaEstimators (e.g. `OneVsOneClassifier`) only accepts `X` and `y` as parameters. Underlying estimators like `SGDClassifier` accept more optional keyword args to this function which are essential for some tasks (e.g. the [`sample_weight`](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn.linear_model.SGDClassifier.fit) parameter is the only way to add weights to training samples in multi-label classification problems).

#### Steps/Code to Reproduce
Here's how I solve a multi-label classification task with a linear SVM

```python
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import SGDClassifier

X = [[0.0, 0.0], [1.0, 1.0], [1.0, 0.0]]
y = [[0], [1], [0, 1]]
y_mlb = MultiLabelBinarizer().fit_transform(y)
sample_weight = [1.0, 0.5, 0.8]
clf = OneVsRestClassifier(SGDClassifier(loss="hinge"))
clf.fit(X, y_mlb)   # unable to pass `sample_weight`
```

see also this [related question](https://stackoverflow.com/q/49534490/599739) on stackoverflow.

#### Expected Results
For regular (single-label) classification tasks, I can pass the `sample_weight` kwarg directly to `SGDClassifier.fit`, but with `OneVsRestClassifier` this is not possible

#### Actual Results
I cannot add weights to my training samples when `OneVsRestClassifier` (or a similar meta-estimator) is used.

#### Feature Request
Please let the `fit` method of `OneVsRestClassifier` (and similar meta-estimators) accept arbitrary kwargs and pass them on to the `fit` method of the wrapped estimator. The same may be useful for `partial_fit` and `score`, though I'm not sure about that.
user: klamann
labels: Needs Decision: Requires decision
issue_10875: state: open
title: [MRG+1] Enable using 2SD scaling in StandardScaler
body: #### Reference Issues/PRs
Fixes #10864

#### What does this implement/fix? Explain your changes.
Enables using 2SD scaling in StandardScaler.

References:
Gelman, A. (2008). Scaling regression inputs by dividing by two standard deviations.
Statistics in medicine, 27(15), 2865-2873.

user: parth1993
labels: Needs Decision: Requires decision
issue_10864: state: open
title: Enable using 2SD scaling in StandardScaler
body: [Gelman (2008)](http://www.stat.columbia.edu/~gelman/research/published/standardizing7.pdf) suggests that scaling by factor of 2SD yields more interpretable results then scaling by 1SD, since it makes both binary and other numerical variables have reasonable scale. Maybe such scaling should be made possible in `StandardScaler` as it is very easy to implement?
user: twolodzko
issue_10863: state: open
title: GMM covariance types examples overly complex / confusing
body: http://scikit-learn.org/dev/auto_examples/mixture/plot_gmm_covariances.html#sphx-glr-auto-examples-mixture-plot-gmm-covariances-py

I don't particularly like the example because it's supervised and uses a train/test split and identification with the original classes. I think it would be better to use a synthetic dataset and just show off the different covariance types.

It also fits the model on 4d data and only shows a 2d projection and that's not super intuitive imho.

Also, the example could be much simplified if we added a "get_covariance" function back to the model. I think we had that in the old GMM. Was there a reason not to add it to the new GMM? In many cases the user wants to be agnostic to the storage format of the covariance matrix, I think.
user: amueller
issue_10862: state: open
title: DictVectorizer.inverse_transform not working properly for boolean types
body: #### Description
Not sure if this is actually a bug or the desired behaviour of DictVectorizer when dealing with booleans. When performing `inverse_transform` on transformed data, if the boolean is `True` the feature is returned with a value of `1.0`, however if the boolean is `False` the feature is not even returned back. This can be quite problematic since the DictVectorizer is essentially removing information about the features when a feature being `False` might actually be needed later on. I'd imagine that the principle behind Vectorizers is that they faithfully transform data rather than removing it.

#### Steps/Code to Reproduce
```
from sklearn.feature_extraction import DictVectorizer
some_data = [{'foo': 2, 'bar': True}, {'foo':10, 'bar': False}]
dv = DictVectorizer(sparse=False)
X = dv.fit_transform(some_data)
dv.inverse_transform(X)
```

#### Expected Results
You would expect `inverse_transform` to return back the data as it was passed in with the same values for booleans.
```[{'foo': 2, 'bar': True}, {'foo':10, 'bar': False}]```

#### Actual Results
What actually happens is that calling `inverse_transform` removes the `'bar':False` item from the feature set. This is quite problematic if you want to chain steps after using a DictVectorizer and are dealing with boolean features.
```[{'bar': 1.0, 'foo': 2.0}, {'foo': 10.0}]```

My suggestion would be to alter how booleans are dealt with within DictVectorizer either by treating them as strings or as booleans in their own right separate from strings and other types.

#### Versions
Darwin-16.7.0-x86_64-i386-64bit
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:04:09) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
NumPy 1.13.3
SciPy 0.19.1
Scikit-Learn 0.20.dev0
user: uwaisiqbal
issue_10799: state: open
title: [MRG] Calculation of information in nats or bits depending on log_base param
body: #### Reference Issues https://github.com/scikit-learn/scikit-learn/issues/10778

#### What does this implement/fix?
Added a parameter log_base to toggle between calculation of information in bits vs nats.
Currently, the calculation of information is only for nats, which is not conventional and confusing. Log_base defaults to 'e' such that existing code will not break. Providing log_base = 2 will give the results in bits units.

#### Any other comments?
There is an SO thread about this [here](https://stackoverflow.com/questions/24686374/pythons-implementation-of-mutual-information)
user: HJLebbink
issue_10796: state: open
title: Reduced-rank regression
body: [Reduced-rank regression](https://www.sciencedirect.com/science/article/pii/0047259X75900421) seems like a simple and fairly well-established technique. It could be implemented minimally by adding a `rank={None, int}` kwarg to Ridge with the svd solver.
user: jona-sassenhagen
issue_10791: state: open
title: Extension request: initialize feature->index mapping for DictVectorizer and CountVectorizer 
body: 
I would like to have the possibility to initialize `DictVectorizer` and `CountVectorizer` with initial feature_name->index dictionary.

This comes in handy when I have a model (say from liblinear) which is trained externally to sklearn, and which I would then want to load and use from within sklearn, or when I interact with external, proprietary feature extractors.

The change to the interface would be adding an optional dict parameter to both constructors. The implementation will be handling `vocabulary_` and `feature_names` accordingly if the dict has non-None value.

Are there any downsides to this which I am missing?

If I make such a PR, will it be accepted?
user: yoavg
issue_10767: state: open
title: [Feature-Request] Add a flag to StratifiedKFold to force classes with only 1 sample in training
body: #### Description

Add a flag to `StratifiedKFold` which ensures each class is present in training set.
For `StratifiedKFold.split`, if some class has only 1 sample, currently this sample might be included in the test split rather than the training split. (`sklearn` does give a warning.)
While for some applications this can be acceptable, a flag which forces classes with a single sample to always be in training can be helpful

#### Steps/Code to Reproduce

```python
from sklearn.model_selection import StratifiedKFold
import numpy as np
X = np.array([0, 1, 2])
y = np.array([0, 0, 1])


skf = StratifiedKFold(n_splits=2, random_state=0, shuffle=True)
for train_index, test_index in skf.split(X, y):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
```

#### Expected Results

There should be some flag in StratifiedKFold, so that atleast 1 element of class `1` is always present in train (in this case, the element at index `2`)
```
Warning: The least populated class in y has only 1 members, which is too few. The minimum number of members in any class cannot be less than n_splits=2.
  % (min_groups, self.n_splits)), Warning)
TRAIN: [0 2] TEST: [1]
TRAIN: [1 2] TEST: [0]
```

#### Actual Results

```
Warning: The least populated class in y has only 1 members, which is too few. The minimum number of members in any class cannot be less than n_splits=2.
  % (min_groups, self.n_splits)), Warning)
TRAIN: [0 2] TEST: [1]
TRAIN: [1] TEST: [0 2]
```
#### Versions

```bash
Linux-4.13.0-36-generic-x86_64-with-Ubuntu-16.04-xenial
Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609]
NumPy 1.14.0
SciPy 1.0.0
Scikit-Learn 0.19.1
```


<!-- Thanks for contributing! -->

user: akhilkedia
issue_10746: state: open
title: Add a "positive" argument to SGDClassifier 
body: `ElasticNet` contains a `positive` argument that constrains coefficients to positive numbers or 0. There is no equivalent in `SGDClassifier`. I would like to attempt to add it.
user: lstmemery
labels: Needs Decision: Requires decision
issue_10745: state: open
title: [WIP] Refactor cd
body: #### Overview
This PR refactors `cd_fast.pyx` module (`linear_module`) which implements coordinate-descent, and make it more transparent and extensible.

The resulting implementation, is [a single function](https://github.com/dohmatob/scikit-learn/blob/refactor-cd/sklearn/linear_model/coordescendant.pyx) which can solve general penalized / constrained multi-task least-squares. regression problems. Out-of-the-box, it supports
  - Gram mode (crucial when p << n) and non-Gram mode (crucial when p >> n)
  - a variety of penalties and constraints, including:
    * L11 penalty (sum of L1 norms), as in multi-task Lasso;
    * L21 penalty (sum of L2 norms), as in G-Lasso;
    * L2INF constraint (L2 constraint on each block), as in vanilla dictionary-learning;
    * L1INF constraint (L1 constraint on each block), e.g as in sparse PCA

The API remains consistent with the legacy implementation via [cd_fast2](https://github.com/dohmatob/scikit-learn/blob/refactor-cd/sklearn/linear_model/cd_fast2.pyx).

Ping @agramfort 

**Other benefits**
* Things like screening rules for LASSO-type problems (inexact Stanford, GapSafe, etc.) and be implemented in a global way (by just modifying the single central cd solver), reducing maintenance complexity
 * The use of cBLAS has been factored out into a [transparent API](https://github.com/dohmatob/scikit-learn/blob/refactor-cd/sklearn/linear_model/blas_api.pxd) which is easier to understand, modify or replace (say, with the modern scipy's [cython_blas]([url](https://docs.scipy.org/doc/scipy/reference/linalg.cython_blas.html)) way)
* Factored the implementation of the dual gap computation into [a single transparent function](https://github.com/dohmatob/scikit-learn/blob/refactor-cd/sklearn/linear_model/utils.pyx#L69).

**Also**
 *  It also supports user-defined penalties (via their prox operators). For example, with this technology, it is straightforward to implement otherwise non-trivial models like:
  - BCD dictionary update in online structured dictionary-learning with general (e.g see Dohmatob el al. "Learning brain regions via large-scale online structured sparse dictionary-learning", In NIPS 2016")

* The same code can be used in implement dictionary updates (once again with both standard and exotic penalties / constraints), via a single appropriate call (another upcoming PR)

**Some very preliminary benchmarks**
![bench](https://user-images.githubusercontent.com/634068/36889289-aaea2166-1df9-11e8-9252-91904e4561bb.png)

* Script: https://github.com/dohmatob/scikit-learn/blob/refactor-cd/benchmarks/bench_lasso_cd_fast2.py


user: dohmatob
issue_10742: state: open
title: whitespace is a terrible separator for feature names in PolynomialFeatures.get_feature_names
body: If your feature names have white spaces in them, it's hard to see which features are interactions right now.
Maybe we should make the separator an option of ``get_feature_names`` or use something like ``*``?
user: amueller
labels: Needs Decision: Requires decision
issue_10725: state: open
title: BUG Median not always being calculated correctly for DecisionTrees in the WeightedMedianCalculator
body: It appears the existing "_WeightedMedianCalculator.update_median_parameters_post_remove_(...)" is not working correctly...  Here is what I am seeing with an illustrated example...

```
y = [0.14038694, 0.4173048, 0.55868983]
sample_weights = [2, 1, 2]
```

in effect, the above is equivalent to:

> 0.14038694, 0.14038694, 0.4173048, 0.55868983, 0.55868983


The process starts with these items being added to the _WeightedMedianCalculator_...  
I have outputted key values after each movement (after _update_median_parameters_post_push_ and after _update_median_parameters_post_remove_ method calls):


`> PUSH, data: 0.4173048;  orig_median: 0.0; new_median: 0.4173048;....`
This tells us that the sample line with y = 0.4173048 was added to a new group.  Since there is only one item this is the same as the median!


`> PUSH;  data: 0.55868983;  orig_median: 0.4173048; new_median: 0.55868983;....`
Next the 0.55868983 sample line is added to the group **(note that this has a weight of 2).**  We now have in effect:
> 0.4174048, **0.55868983**, 0.55868983
 
so the new median becomes 0.55868983 as expected.


`> PUSH;  data: 0.14038694;  orig_median: 0.55868983; new_median: 0.4173048;....`
Next the 0.14038694 sample line is added to the group.  We now have in effect:
> 0.14038694, 0.14038694, **0.4173048**, 0.55868983, 0.55868983

so the new median becomes 0.4173048 as expected.


`> POP/REMOVE;   data: 0.4173048;  orig_median: 0.4173048; new_median: 0.55868983;....`

The data equated to the sample line 0.4173048 is removed.  We now have:
> 0.14038694, **0.14038694, 0.55868983**, 0.55868983

The new median looks incorrect!  Should it not be:
= (0.14038694 + 0.55868983) / 2
= 0.349538385
?

Steps to reproduce:
Unfortunately it is difficult to provide a python script to demonstrate the above, as it requires debugging private values but can be achieved this way:
/tree/utils.pyx -> WeightedMedianCalculator class:

```
    cdef int remove(self, DOUBLE_t data, DOUBLE_t weight) nogil:
        """Remove a value from the MedianHeap, removing it
        from consideration in the median calculation
        """
        cdef int return_value
        cdef DOUBLE_t original_median

        if self.size() != 0:
            original_median = self.get_median()

        #FOR DEBUG:
        cdef int original_k
        cdef DOUBLE_t original_sum_w_0_k
        cdef DOUBLE_t original_total_weight
        original_k = self.k
        original_sum_w_0_k = self.sum_w_0_k
        original_total_weight = self.total_weight

        return_value = self.samples.remove(data, weight)
        self.update_median_parameters_post_remove(data, weight,
                                                  original_median)

        with gil:
            print (" ")
            print ("POP/REMOVE, data: " + str(data)
                + ";  orig_median: " + str(original_median) 
                + "; new_median: " + str(self.get_median()) 
                + ";  size: " + str(self.size())
                + ";  weight: " + str(weight)
                + ";  orig_total_weight: " + str(original_total_weight)
                + ";  self.total_weight: " + str(self.total_weight)
                + ";  orig_k: " + str(original_k)               
                + ";  self.k: " + str(self.k)
                + ";  orig_sum_w_0_k: " + str(original_sum_w_0_k)
                + ";  self.sum_w_0_k: " + str(self.sum_w_0_k)
                )

        return return_value
```
```
        cdef int push(self, DOUBLE_t data, DOUBLE_t weight) nogil except -1:
        """Push a value and its associated weight to the WeightedMedianCalculator

        Return -1 in case of failure to allocate memory (and raise MemoryError)
        or 0 otherwise.
        """
        cdef int return_value
        cdef DOUBLE_t original_median

        #FOR DEBUG:
        cdef int original_k
        cdef DOUBLE_t original_sum_w_0_k
        cdef DOUBLE_t original_total_weight
        original_k = self.k
        original_sum_w_0_k = self.sum_w_0_k
        original_total_weight = self.total_weight

        if self.size() != 0:
            original_median = self.get_median()
        # samples.push (WeightedPQueue.push) uses safe_realloc, hence except -1
        return_value = self.samples.push(data, weight)
        self.update_median_parameters_post_push(data, weight,
                                                original_median)

        with gil:
            print (" ")
            print ("PUSH, data: " + str(data) 
                + ";  orig_median: " + str(original_median) 
                + "; new_median: " + str(self.get_median()) 
                + ";  size: " + str(self.size())
                + ";  weight: " + str(weight)
                + ";  orig_total_weight: " + str(original_total_weight)
                + ";  self.total_weight: " + str(self.total_weight)
                + ";  orig_k: " + str(original_k)               
                + ";  self.k: " + str(self.k)
                + ";  orig_sum_w_0_k: " + str(original_sum_w_0_k)
                + ";  self.sum_w_0_k: " + str(self.sum_w_0_k)
                )

        return return_value
```

After compilation, calling this script:

```
from sklearn.tree import DecisionTreeRegressor

train_y = [0.4173048,0.55868983,0.14038694]
train_X = [[ 15., 9.],
           [ 19., 35.],
           [ 40., 54.]]
sample_weight = [1,2,2]

depth = 1

wineTree = DecisionTreeRegressor(max_depth=depth, criterion='mae', random_state=1)
wineTree.fit(train_X, train_y
    , sample_weight=sample_weight
    )
```

I think I know where the problem lies and how to fix this issue, but first wanted to ensure this behaviour isn't desired especially considering sample_weights could be represented as floats etc?



user: JohnStott
issue_10706: state: open
title: Inconsistent behavior in averaging choices micro and samples.
body: For the multi-class case, precision, recall and f-score with ``micro``  all produce accuracy, while with ``samples`` they produce an error.
That seem inconsistent. Using the definitions in the docs, they should also all be accuracy, I think.

I think I'd propose to deprecate ``micro`` averaging for multiclass.

The docs actually give an example of micro-average recall for multiclass, which is really weird imho.
user: amueller
issue_10684: state: open
title: Stratify option for learning_curve
body: Currently `learning_curve` doesn't support stratified splits of training data, i.e. when we take only part of training data, this part may contain any ratio of class labels. In case of big class imbalance and small parts of training data (i.e. small first value of train_size parameter), this often leads to the absence of instances belonging to rare class(es), which, in turn, leads to the impossibility of classifier training.
I can't provide actual data I used, but it is pretty easy to simulate this with very small mock data sets, let me know if I should do that.

I implemented this functionality for myself by adding 1 parameter and 6 lines of code to learning_curve (using `StratifiedShuffleSplit` instead of `train[:n_train_samples]` on [this line](https://github.com/scikit-learn/scikit-learn/blob/0195facbcb7986f6b578ccafa1c61afa7ee8ae08/sklearn/model_selection/_validation.py#L1123)) and will be happy to prepare pull request with tests and all needed stuff, but would firstly like to know your thoughts on this. 
There were mentions of this functionality previously, see [PR 2701 ](https://github.com/scikit-learn/scikit-learn/pull/2701) and [PR 7506](https://github.com/scikit-learn/scikit-learn/pull/7506) - but both times it was kind of decided that shuffling would be enough: 'kind of', because no actual discussion on this topic happened.
user: astrakhantsev
issue_10624: state: open
title: MAINT Use Cython memoryviews
body: Currently a [fair amount of  Cython code](https://github.com/scikit-learn/scikit-learn/search?p=3&q=np.ndarray&type=&utf8=%E2%9C%93) uses the [numpy buffer interface](http://cython.readthedocs.io/en/latest/src/tutorial/numpy.html#efficient-indexing).

 When possible, it could be worth migrating some of it to the [typed memoryviews Cython interface](http://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html) that has lower overhead and could result in some performance improvements [[1](http://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html#comparison-to-the-old-buffer-support)], [[2](https://jakevdp.github.io/blog/2012/08/08/memoryview-benchmarks/)].  This would also allow to release GIL more frequently which would be positive for parallel computations with the threading backend.

I'm not saying that it makes sense to use memoryviews everywhere, and there are subtle differences between the two (possibly related https://github.com/scikit-learn/scikit-learn/issues/7059#issuecomment-234330147). Just opening this issue to track progress.

 For instance, [ `sklearn/utils/sparsefuncs_fast.pyx`](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/sparsefuncs_fast.pyx) could be a place to start.
user: rth
issue_10613: state: open
title: Logistic regression prior adjustment during prediction when class_weight="balanced"
body: I quickly glanced through the code and did not find any intercept adjustment during test (prediction) time when the estimator is initialized as ``LogisticRegression(class_weight="balanced")`` and the docs don't suggest this either. Unless I am missing something, there does not appear to be any readjustment of the class-priors after an "artificially" balanced prior is used to train the model. If it is not implemented, might this be something worth adding?

References:
1. https://stats.stackexchange.com/questions/117592/logistic-regression-prior-correction-at-test-time
2. https://stats.stackexchange.com/questions/6067/does-an-unbalanced-sample-matter-when-doing-logistic-regression
3. https://stats.stackexchange.com/questions/67903/does-down-sampling-change-logistic-regression-coefficients
user: ashimb9
issue_10612: state: open
title: Added support for not normalising OneVsRestClassifier predict_proba result
body: #### Reference Issues/PRs
None

#### What does this implement/fix? Explain your changes.
I added support for not normalizing the predict_proba result, and returning it as it is, even if the data is not multilabel. This is useful for implementing a rejection threshold.

#### Any other comments?
I can also add thresholding in ovrclassifier or somewhere else to reject classifications results, but I didn't see anything else like that in scikit, so I don't know if it's a wanted feature.

user: EmreAtes
labels: Needs Decision: Requires decision
issue_10605: state: open
title: Mean centring issues in PCA/PLS when used in a Pipeline
body: #### Description
The scikit-learn implementations of PCA ( sklearn.decomposition.PCA) and PLS algorithms (sklearn.cross_decomposition._PLS objects, exemplified here with PLSRegression) automatically perform mean centering as part of their fit method. 

This can lead to unnecessary or even problematic duplicated centering when using these objects inside Pipelines, especially with RobustScaler or scalers that do not center using the arithmetic mean vector. 

In summary, The center_/mean_ centering applied by the scaler object is further adjusted by the .mean_/.x_mean_ stored inside the objects during .fit.

I guess having a center option in the __init__ or fit method for those cases (and kept as True by default) would fix this (always set to False when using a scaler in a pipeline), as well as allow more flexible use of these algorithms (ie, explore double-row/column centring). 

#### Steps/Code to Reproduce
This can be reproduced by simply using a pipeline with PCA/PLS regression and any scaler before. I have quickly adapted an example with the wine dataset to showcase the double scaling with RobustScaler.

```python
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, RobustScaler
from sklearn.decomposition import PCA
from sklearn.cross_decomposition import PLSRegression
from sklearn.datasets import load_wine
from sklearn.pipeline import make_pipeline

RANDOM_STATE = 42

features, target = load_wine(return_X_y=True)

# Make a train/test split using 30% test size
X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.30, random_state=RANDOM_STATE)

# Fit to data - PCA.
unscaled_clf = make_pipeline(PCA(n_components=2))
unscaled_clf.fit(X_train)

# Fit to data - PLSRegression
unscaled_pls = make_pipeline(PLSRegression(2, copy=False, scale=False))
unscaled_pls.fit(X_train, y_train)

# Fit to data - StandardScaler and PCA pipeline.
std_clf = make_pipeline(StandardScaler(), PCA(n_components=2))
std_clf.fit(X_train)

# Fit to data - StandardScaler and PLSRegression pipeline.
std_pls = make_pipeline(StandardScaler(), PLSRegression(2, copy=False, scale=False))
std_pls.fit(X_train, y_train)

# Fit to data - RobustScaler and PCA pipeline.
rob_clf = make_pipeline(RobustScaler(), PCA(n_components=2))
rob_clf.fit(X_train, y_train)

# Fit to data - RobustScaler and PLSRegression pipeline.
rob_pls = make_pipeline(RobustScaler(), PLSRegression(2, copy=False, scale=False))
rob_pls.fit(X_train, y_train)

# PCA 
# Mean center vector calculated when using .fit
print(unscaled_clf.named_steps['pca'].mean_)

# Pipeline StandardScaler + PCA
# Mean center vector from the scaler 
print(std_clf.named_steps['standardscaler'].mean_)
# Mean center vector inside PCA
print(std_clf.named_steps['pca'].mean_)

# Pipeline - RobustScaler + PCA
print(rob_clf.named_steps['robustscaler'].center_)
print(rob_clf.named_steps['pca'].mean_)

# PLSRegression
print(unscaled_pls.named_steps['plsregression'].x_mean_)

# Pipeline - StandardScaler + PLSRegression
print(std_pls.named_steps['standardscaler'].mean_)
print(std_pls.named_steps['plsregression'].x_mean_)

# Pipeline - RobustScaler + PLSRegression
print(rob_pls.named_steps['robustscaler'].center_)
print(rob_pls.named_steps['plsregression'].x_mean_)

```
#### Results
For the Pipelines with StandardScaler, the effect is quite "benign", as these scalers use the same 
criteria. The differences are close to numerical tolerance. However, when using RobustScaler, the second "unexpected" scaling is no longer negligible...

#### Versions
Please run the following snippet and paste the output below.
Linux-4.13.0-32-generic-x86_64-with-debian-stretch-sid
Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19) 
[GCC 7.2.0]
NumPy 1.14.0
SciPy 1.0.0
Scikit-Learn 0.19.1
user: Gscorreia89
issue_10604: state: open
title: [MRG]Clusters-Class auto-match
body: Since I needed to calculate accuracy from a clustering result in my research, I think it could be really useful to have it on the library depending on the task.
Specially useful when the task being implemented involves different sets of classes for each new dataset.
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

#### What does this implement/fix? Explain your changes.
The class_cluster_match(y_true, y_pred) translates clusters found through clustering methods, generally given in numbers to the corresponding best possible match of true class labels, generally strings.

#### Any other comments?
The max_main_diagonal was implemented by user Paul Panzer at https://stackoverflow.com/questions/48511584/how-to-efficiently-make-class-to-cluster-matching-in-order-to-calculate-resultin.
<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: LPugens
issue_10603: state: open
title: Support standard data science use-case
body: I brought this up in a couple of places, but I think it's important enough to have it being tracked as a separate issue. We often use the tenet "Simple things should be simple, complex things should be possible." However, right now, we don't support what I think it the most simple and trivial data-science use case with tabular data.

By default, in the data science world, you have a mix of continuous and discrete variables, and there are missing values. Right now, it's impossible to do cross-validation on a dataset like that.

With #9012 in the next release this would be possible using something like

```python
categorical = X.dtypes == object
preprocess = make_column_transformer(
    (make_pipeline(Imputer('median'), StandardScaler()), ~categorical),
    (make_pipeline(CategoricalEncoder(encoding='ordinal'),
    Imputer(method='mosts_frequent'), CategoricalEncoder()), categorical))
model = make_pipeline(preprocess, LogisticRegression())
```

For many people, this is the most simple and basic use-case of scikit-learn. Learning logistic regression on a tabular dataset. This is not simple.

I think we should strive to support this as a first-class use-case - because it is.

Right now, everywhere I look I see custom implementations to fix this issue. It would be great if we could solve this problem here once and for all. It's really actually quite simple.

In https://github.com/scikit-learn/scikit-learn/issues/2888#issuecomment-363598565 @jnothman said

> This sort of sounds like it belongs in some sklearn-building-blocks library rather than the core library, but I see why you'd want it here. From a "everything is an array" traditional scikit-learn perpsective, it looks like automl magic. From an R "everything is columns of different data types" perspective it looks like nothing else should have been considered.

In a sense I'd be happy having a building blocks library. But the question is: what else would there be in it? The only thing that literally every person I talk to needs and that we don't have is being able to handle categorical variables in a reasonable way.

EDIT:
```python
categorical = X.dtypes == object
preprocess = make_column_transformer(
    (make_pipeline(Imputer(), StandardScaler()), ~categorical),
    (make_pipeline(Imputer(strategy='constant', fill_value='NA'),
     OneHotEncoder(handle_unknown='ignore')), categorical))
model = make_pipeline(preprocess, LogisticRegression())
```
is now possible and mostly does what I wanted it to do.
Finding the correspondence of the coefficients to the original features is still challenging and this is still quite a bit of code. Having a ``BasicPreprocessor`` class that does the above might be useful.
user: amueller
issue_10600: state: open
title: Automatically force_all_finite=False in non-initial warm_start runs
body: For estimators that have a `warm_start` option, we generally expect that the same data is passed in for each call to `fit` as was done in the first. I think we can save a little on runtime by using `check_array` with `force_all_finite=False` in the case that `warm_start=True` and the model has already been fitted.
user: jnothman
issue_10597: state: open
title: [MRG] Adding LinearKernel to GaussianProcesses
body: #### Reference Issues/PRs

Fixes #8359
Closes  #8373  

#### What does this implement/fix? Explain your changes.

The Gaussian Process code is still missing a basic implementation of a linear kernel. This project was started by @dalmia, but it seems some of the reviewer comments were never addressed. I've made an attempt at addressing them and have been using it for my own purposes. It would be nice to make this standard functionality.

#### Any other comments?

- [x] Remove the varaiance parameters 
- [x] Replace `_check_offset` with `_check_length_scale`
- [x] Make sure tests pass

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: jboes
issue_10590: state: open
title: Regarding usage of datetime64[ns] dtype in some estimators
body: This question is related to [this stackoverflow question](https://stackoverflow.com/questions/35439723/using-pandas-timestamp-with-scikit-learn)

Why is that the some estimators are allowed to work with numpy datetime64[ns] ? Is it proper usage? If yes, then how are the calculations done internally for that column? 

#### Steps/Code to Reproduce
```
import pandas as pd
import numpy as np
data_array = [['11/18/2017', '453'],
              ['11/19/2017', '453'],
              ['11/20/2017', '453'],
              ['11/21/2017', '201'],
              ['12/11/2017', '202'],
              ['12/13/2017', '2045'],
              ['12/13/2017', '347'],
              ['12/14/2017', '981']]

dataset = pd.DataFrame(data_array, columns=['date', 'sales'])
date = pd.DataFrame(pd.to_datetime(dataset['date']))
sale = pd.DataFrame(dataset['sales'])

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(date,sale, test_size = 0.25)

from sklearn.base import RegressorMixin
from sklearn.utils.testing import all_estimators
regressors=[est for est in all_estimators() if issubclass(est[1], RegressorMixin)]

#from sklearn.tree import DecisionTreeRegressor
#regressor = RandomForestRegressor(n_estimators = 300)
from sklearn.utils.validation import check_array
print(check_array(X_train).dtype)

for regressor in regressors:
    print(regressor)
    reg = regressor[1]()

    try:
        reg.fit(X_train,y_train)
        print(reg.predict(X_test))
        print(reg.score(X_test, y_test))
        print('Success')
    except (TypeError, ValueError) as e:
        print(e, ' -> ',regressor[0])
```


#### Versions
Linux-3.16.0-77-generic-x86_64-with-Ubuntu-14.04-trusty
('Python', '2.7.6 (default, Nov 23 2017, 15:49:48) \n[GCC 4.8.4]')
('NumPy', '1.14.0')
('SciPy', '1.0.0')
('Scikit-Learn', '0.19.1')


I'm sorry if this has been discussed before here. I searched for datetime64 in the issues and could not find anything. 

<!-- Thanks for contributing! -->

user: vivekk0903
labels: Needs Decision: Requires decision
issue_10584: state: open
title: Improve documentation: consistent scoring functions 
body: #### Abstract
Improve the documentation [3.3. Model Evaluation](http://scikit-learn.org/stable/modules/model_evaluation.html). Give advice to use strictly proper scoring functions.

#### Explanation
The documentation of scikit-learn is amazing and a strong argument for usage (and fame). I think that one could get a bit lost by choosing _the right_ scoring function or metric out of the many options for model evaluation. There are some influential papers which advocate the usage of **strictly proper scoring functions**:

1. [Making and Evaluating Point Forecasts](https://arxiv.org/abs/0912.0902), [alt. link](https://www.bundesbank.de/Redaktion/EN/Downloads/Bundesbank/Research_Centre/Conferences/2012/2012_06_01_eltville_11_gneiting_paper.pdf)

2. [Strictly Proper Scoring Rules, Prediction,
and Estimation](https://doi.org/10.1198/016214506000001437), [alt. link](https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf)

For classification and regression, most of the time one is interested in the mean functional of the distribution of the target variable `y`. Then, the scoring function used to compare the predictive power of different models (like when choosing the regularization strength via cross validation) should be strictly consistent for the mean functional.

#### Examples
For binary classification knowing the mean of `y` is equivalent to knowing the whole distribution of `y`. Brier (squared error) and logistic (log) loss are strictly consistent for the mean. Accuracy and 0-1 loss are only consistent, but not strictly consistent (they are strictly consistent for the mode which is less informative than the mean for classification). ROC, precision, recall etc. are not consistent for the mean, afaik.

For regression, Bregman functions (eq. (18) of 1.) are strictly consistent for the mean. For targets `y` on the whole real numbers, the squared error is one example. For positive-valued targets `y`, the squared error (b=2), Gamma deviance (b=0) and Poisson deviance (b=1) are examples, see eq. (20) of 1.

Maybe one should check the many metrics already provided by scikit-learn, if they are (strictly) consistent for a certain functional and if they are equivalent to one another.

#### Disclaimer
I do not know the authors of the cited papers and I'm not pushing my own research, just my opinion on how to improve this fantastic library 😏  
user: lorentzenchr
issue_10573: state: open
title: In macro-averaged multiclass precision, and recall, the default set of labels should come only from the target outputs (y_true) not from the predicted outputs (y_pred)
body: #### Description
In macro-averaged multiclass precision, and recall, the default set of labels should come only from the target outputs (y_true) not from the predicted outputs (y_pred).

The target outputs come from the test set.
I believe it is a solid assumption is that the classes we are interested in (at all) are the classes present in the the test set.
We don't want to average over classes that we are not interested in.
In the case where a label occurs only in the predicted argument

Trying to calculate recall on classes that only occur in the test set is (as the warning say) ill defined anyway.
This would make the behavior consistent with `average=weighted` , which only averages over true classes.

More consideration at the bottom, though it is a bit long winded even by my standards.

#### Steps/Code to Reproduce /  Actual Results

```python
In [1]: from sklearn.metrics import precision_recall_fscore_support

In [2]: precision_recall_fscore_support(list("aabbb"), list("aabbc"), average='macro')
/usr/local/lib/python2.7/dist-packages/sklearn/metrics/classification.py:1137: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples.
  #!/usr/bin/python
Out[2]: (0.66666666666666663, 0.55555555555555547, 0.59999999999999998, None)

#### Expected Results
```python
In [1]: from sklearn.metrics import precision_recall_fscore_support

In [2]: precision_recall_fscore_support(list("aabbb"), list("aabbc"), average='macro') 
Out[2]: (1.0, 0.83333333333333326, 0.90000000000000002, None)
```

Possibly with a `"Warning:  labels ['c'] occurs in predictions but not in targets. It is not counted. To count it, define labels argument manually. However, this will cause Recall and F-Score to be ill-defined."`

Also reasonable would be for it to throw an error,
and require the user to specify the labels one way or the other.
Either as `labels=unique_labels(y_true, y_pred)` (current behavior),
or as `labels=unique_labels(y_true)` (what I suggest is the more commonly desired behavior)

or as a final option, 
not changing the behavior but adding to the current warning message: `you may wish to only consider labels from y_true. To do this use "labels=labels=unique_labels(y_true)`

There are matching changes that would come to `average=None` from this change as well.

#### Versions
I first encounter this in Scikit-Learn 0.14.1'.
(turned out I was using an old version by mistake),
but it is unchanged when I updated to 0.19.1

```python
In [1]: import platform; print(platform.platform())
Linux-3.16.0-4-amd64-x86_64-with-debian-8.10

In [2]: import sys; print("Python", sys.version)
('Python', '2.7.9 (default, Jun 29 2016, 13:08:31) \n[GCC 4.9.2]')

In [3]: import numpy; print("NumPy", numpy.__version__)
('NumPy', '1.9.3')

In [4]: import scipy; print("SciPy", scipy.__version__)
('SciPy', '0.15.1')

In [5]: import sklearn; print("Scikit-Learn", sklearn.__version__)
('Scikit-Learn', '0.19.1')
```

### Extended Motivation.

Now, in normal supervised multi-class ML circumstances you will not have a label that appears in the *predicted* output but not the *target* output.

There are times when you can, (I am working with one in  https://github.com/oxinabox/NovelPerspective)
(though the question then becomes is Macro Averaged P/R/F1 a good metric for these? Maybe not)

Anyway, we must assert that we *are* in such a circumstance, as otherwise, why have we bothered to search both target and predicted output.


Consider:
These measures come from information extraction.
They are binary measures.
The precision is "Of all the times I found LABEL, how many of them were where they should have been " `Prec = tp/(tp+fp)`
The Recall is precision is "Of all times I should have found LABEL, how often did I actually find it?" `Recall = tp/(tp+fn)`


When define a test set (i.e. the target set), the classes in it must be the ones that matter. It is the ground truth.

We wouldn't make a test set that didn't have a class that matters (or would we?)

When we same Macro averaging,
we are thinking:
For each class rewrite the outputs so that it is binary as to (Found, Not Found) for the class in question.
And calculate the information extraction precision/recall for each class.
Then average those.

In that binary view, you never see the classes from the *predicted output*.
Just "Not Found" for every class in the *target* output.

Because it is the *target* output classes that you are asking questions about -- that is why they are in the test set.


Now consider the math
for some class that *is* only in the *predicted* output

lets say it occurs `N` times.
then `tp=0` and `fn=0` since it has no true positives or false negatives since it never occurs.

and `fp=N` since any time it occurs is a false positive.

Its (binary) precision will be 0. `tp/(tp +fp)=0/(0+N)`

its (binary) recall will be Undefined:
`tp/(fn+tp) = 0/(0+0)`
its F1 will thus also be undefined.

Sklearn gives a warning and sets the binary recall and F1 to zero.

This in turn decreases the (overall) macro average since it adds a class that scores 0 for precision and recall.

user: oxinabox
issue_10539: state: open
title: Inconsistent density estimator sampling APIs
body: The [KernelDensity sample() method](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KernelDensity.html#sklearn.neighbors.KernelDensity.sample) provides an optional `random_state` arg, to allow samples to be generated reproducibly.

However, the similar [MixtureBase sample() method](http://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html#sklearn.mixture.GaussianMixture.sample) does not have the same API, leading to an inconsistency between density estimator samples that should ideally be interchangeable.

MixtureBase sample() instead uses its internal random_state (optionally passed to its constructor), which does allow reproducible samples to be generated (independently of random number generation within the implementation) using the pattern:
```
fit = mixture.GaussianMixture(...).fit(X)
...
fit.random_state = ...
fit.samples(...)
```

I propose to add an optional `random_state` arg to `MixtureBase.sample` that is used instead of the internal random state when provided.  If others agree, I can provide a simple PR.  This change would not add new capability (given the pattern above) but instead facilitates best practices for reproducible random numbers.

Note that, even with this change, the KernelDensity and MixtureBase sampling APIs are not strictly compatible since MixtureBase samples return labels in addition to the generated samples.  This could be an argument against making this change.
user: dkirkby
labels: Needs Decision - API: Requires decision regarding API
issue_10537: state: open
title: Make ColumnTransformer work on dask arrays?
body: The new `ColumnTransformer` (not merged yet, https://github.com/scikit-learn/scikit-learn/pull/9012) would also be useful in `dask_ml` context when using estimators that can handle dask objects, and it also *almost* works for them. 
Main bottleneck is the concatenation of the results of the different transformers.

With a relatively small patch to the stacking code of `ColumnTransformer.transform` (where we already check for sparse matrices), it works on my test cases:

```patch
         ....
         Xs = list of output of different transformers

         if any(sparse.issparse(f) for f in Xs):
             Xs = sparse.hstack(Xs).tocsr()
+        elif all(hasattr(f, '__dask_graph__') for f in Xs):
+            import dask.array as da
+            Xs = da.hstack(Xs)
         else:
             Xs = np.hstack(Xs)
         return Xs

```

(Note the above is not fully complete as it only works for dask arrays, not dask dataframes)

Possibilities here would be to a) not care about this for now (and dask_ml can override the parts that don't work), b) make it easier for dask_ml to subclass ColumnTransformer by putting the above in a separate method like `_stack_results` that can be overridden by dask_ml c) have a sklearn utils function that is actually dask-aware (the above patch, similar to how the safe_indexing that is pandas aware) so the sklearn version works out of the box and the ColumnTransformer does not need to be partly duplicated in dask_ml, d) ...

We don't necessarily have to decide directly on what to do here, but it might be something we could start thinking about. Also more in general (eg Gaël had the idea of sharing a single file with dispatch functions for such typical things), although this is one is very specific for the ColumnTransformer. 




user: jorisvandenbossche
issue_10530: state: open
title: Misleading kernel PCA example
body: The only text accompanying this [kernel PCA example](http://scikit-learn.org/stable/auto_examples/decomposition/plot_kernel_pca.html) is:

> This example shows that Kernel PCA is able to find a projection of the data that makes data linearly separable.

However, this example is very fine tuned and probably gives most readers a misleading impression.  To demonstrate this, the following function reproduces the example's projection to a 2D latent space (lower left plot):
```
def kpca_demo(factor=.3, gamma=10, seed=0):
    np.random.seed(seed)
    X, y = make_circles(n_samples=400, factor=factor, noise=.05)
    kpca = KernelPCA(kernel="rbf", gamma=gamma)
    X_kpca = kpca.fit_transform(X)
    # Plot projection into the 2D latent space.
    reds = y == 0
    blues = y == 1
    plt.scatter(X_kpca[reds, 0], X_kpca[reds, 1], c="red",
                s=20, edgecolor='k')
    plt.scatter(X_kpca[blues, 0], X_kpca[blues, 1], c="blue",
                s=20, edgecolor='k')
```
With the defaults args, the example is reproduced exactly and the separation is very clear:
```
kpca_demo()
```
![original](https://user-images.githubusercontent.com/185007/35350732-b1385074-00f3-11e8-96d8-fb27b7c51364.png)

However, small changes to any of these args reveal that this nice result is not at all typical, and you are more likely to find a latent space with a similar nonlinearity to the original data:
```
kpca_demo(seed=2) # was seed=0
```
![change_seed](https://user-images.githubusercontent.com/185007/35350739-b4338f78-00f3-11e8-9916-4ec1338cf4a4.png)
```
kpca_demo(factor=.31) # was factor=.3
```
![change_factor](https://user-images.githubusercontent.com/185007/35350755-c038b6e0-00f3-11e8-9fb4-6de6672eac02.png)
```
kpca_demo(gamma=10.2) # was gamma=10
```
![change_gamma](https://user-images.githubusercontent.com/185007/35350758-c5157a0e-00f3-11e8-890b-41c7bc989818.png)

To improve the pedagogical value of this example, I suggest either finding a more robust demonstration of kernel PCA or else commenting that some fine tuning is generally required to achieve linear separation.
user: dkirkby
issue_10525: state: open
title: Improve RidgeCV exception message when called with `store_cv_values=False`
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
When RidgeCV is called with default params and I try to access cv_values_ it fails with:

`AttributeError: 'RidgeCV' object has no attribute 'cv_values_'`

I think it should instead raise an error for letting the user know to call RidgeCV with `store_cv_values=True`. This can be achieved using `check_is_fitted` and then raising AttributeError with an appropriate message.  

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

```
import numpy as np

from sklearn.datasets import make_regression
from sklearn.linear_model import RidgeCV

alpha_grid = np.linspace(0.1, 1, 10)
X, y = make_regression(n_features=3, effective_rank=2, noise=10)
clf = RidgeCV(alphas=alpha_grid)
clf.fit(X, y)
print(clf.cv_values_)
```


#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
`AttributeError: Call 'RidgeCV' with 'store_cv_values=True'`
#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
`AttributeError: 'RidgeCV' object has no attribute 'cv_values_'`
#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
Windows-7-6.1.7601-SP1
('Python', '2.7.12 |Anaconda custom (32-bit)| (default, Jun 29 2016, 11:42:13) [MSC v.1500 32 bit (Intel)]')
('NumPy', '1.13.1')
('SciPy', '1.0.0')
('Scikit-Learn', '0.19.1')

<!-- Thanks for contributing! -->

user: maykulkarni
issue_10498: state: open
title: Add sklearn.dataset.fetch_bbc and sklearn.dataset.fetch_bbcsport
body: Here is the source information for those datasets:

 http://mlg.ucd.ie/datasets/bbc.html

It would be great to have the loaders return both the matrix market format (preprocessed token frequencies) and the raw texts as attributes of the `Bunch` object.

The docstring of those functions should mention the usage restrictions and citation requirements as stated on the above website.

If you are new to scikit-learn, have a look at the source code of the `sklearn.datasets.fetch_20newsgroups` and other `fetch_*` functions from the datasets module and their tests in `sklearn/datasets/tests` for inspiration.

Those datasets could be used in the tutorial and examples instead of 20 newsgroups (or maybe in addition to it). I find the BBC datasets better for teaching text classification than 20 newsgroups.

Here is a blog post that shows the kind of results and gives some insightful analysis of the model performance:

https://cloud.google.com/blog/big-data/2018/01/problem-solving-with-ml-automatic-document-classification
user: ogrisel
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_10478: state: open
title: [MRG] Fixes #10470: FeatureHasher ordered feature list
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
Fixes #10470 
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
- [x] Make FeatureHasher provide get_feature_names
- [x] Test changes
#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: maykulkarni
labels: Needs Decision: Requires decision
issue_10470: state: open
title: FeatureHasher cant return order of found feature_names
body: It would be nice if the `FeatureHasher` would be able to return order of found `feature_names` in the given dicts, so you know which column belongs to which feature, just like the `DictVectorizer` does.

```py
from sklearn.feature_extraction import FeatureHasher
h = FeatureHasher(n_features=4)
D = [{'dog': 1, 'cat':2, 'elephant':4},{'dog': 2, 'run': 5}]`
f = h.fit_transform(D)`
f.toarray()
# array([[ 0., -1., -4.,  2.], [-5., -2.,  0.,  0.]])

h.get_feature_names()
# ['run','dog','elephant','cat']
```
user: dreizehnutters
issue_10453: state: open
title: Pandas Interoperability section in User Guide
body: I think the User Guide should have an additional chapter on Interoperability. Its first section should be on Pandas. It should cover:

1. That basically all estimators support DataFrames as long as the data is all numeric, by takings its `.values`. but this will result in a copy, so providing a numpy array is more efficient
2. Some meta-estimators (notably model selection and pipeline tools) will pass a dataframe along as-is to nested estimators.
3. Sparse DataFrames are not automatically converted to scipy.sparse matrices
4. The mismatched column names issue #7242 (perhaps one day we'll warn about this)
5. How to put index back on your data after transformation / prediction (see #5523)
6. Dealing with categorical variables (see CategoricalEncoder and glossary) and missing values (see glossary)
7. Dealing with heterogeneous data (see example on this topic; ColumnTransformer when merged)
8. Constructing feature names for the output of transformers is an open problem, though `get_feature_names` may help; some things are not well described with names
8. see also sklearn-pandas if it's still going to be relevant in 0.20

Ping @dukebody, @thebeancounter, @pjcpjc, @naught101
user: jnothman
issue_10416: state: open
title: check global random state is never required with lgtm.com
body: We should have lgtm.com alerts for cases where we might be using global random state, leading to non-determinism:

* alert if a callable which takes `random_state` is called and `random_state` is not passed
* alert if np.random.something-other-than-RandomState is called
* alert if random.something-other-than-Random is called

(Note this will not check whether the C random number generator is used.)

Ping @markshannon in case you feel like helping us out!
  
user: jnothman
issue_10372: state: open
title: partial_dependence.py _grid_from_X ignores percentile specification unexpectedly
body: 
#### Description
partial_dependence.py _grid_from_X ignores percentile specification if the number of unique values in feature is less than the grid_resolution

#### Steps/Code to Reproduce
``` python
    for col in range(X.shape[1]):
        uniques = np.unique(X[:, col])
        if uniques.shape[0] < grid_resolution:
            # feature has low resolution use unique vals ### <--- this is  unexpected behavior
            axis = uniques
        else:
            # create axis based on percentiles and grid resolution
            axis = np.linspace(emp_percentiles[0, col],
                               emp_percentiles[1, col],
                               num=grid_resolution, endpoint=True)
        axes.append(axis)
```
#### Expected Results
I think this is expected behavior.

``` python
    for col in range(X.shape[1]):
       # unique values within the empirical percentiles
        uniques = np.unique(X[(X[:, col] >= emp_percentiles[0, col]) & (X[:, col] <= emp_percentiles[1, col]), col])
        if uniques.shape[0] < grid_resolution:
            # reduce the grid_resolution if the number of unique values is smaller
            grid_resolution = len(uniques)
        else:
            # create axis based on percentiles and grid resolution
            axis = np.linspace(emp_percentiles[0, col],
                               emp_percentiles[1, col],
                               num=grid_resolution, endpoint=True)
        axes.append(axis)
```
#### Actual Results

#### Versions
Linux-4.3.5-smp-809.21.0.0-x86_64-with-debian-testing-unstable
('Python', '2.7.6 (default, Aug  4 2016, 14:43:59) \n[GCC 4.8.x-google 20131105 (prerelease)]')
('NumPy', '1.11.1')
('SciPy', '1.0.0')
('Scikit-Learn', '0.19.1')


user: sungminkim31
labels: Needs Decision: Requires decision
issue_10368: state: open
title: RFECV float (percentage) step does not reduce features in each iteration as described
body: Hi scikit-learn dev team,

Apologies if I am not understanding how the step feature of RFECV works.  With RFECV, when you specify an integer step, you see in verbose mode that it steps down by exactly the number of features until zero, e.g. step=100:
```
...
Fitting estimator with 1013 features.
Fitting estimator with 913 features.
Fitting estimator with 813 features.
Fitting estimator with 713 features.
Fitting estimator with 613 features.
Fitting estimator with 513 features.
Fitting estimator with 413 features.
Fitting estimator with 313 features.
Fitting estimator with 213 features.
Fitting estimator with 113 features.
Fitting estimator with 13 features.
```
But with a float (percentage) step it doesn't do what you'd expect.  If I specify a step=0.1, I would expect it to reduce the number of features by exactly 10% in each iteration such that it's reducing by fewer and being being more fine-tuned as the number of features decreases.  But it seems to be doing quite the opposite where it removes more features than 10% as it gets closer to zero:
```
Fitting estimator with 54613 features.
Fitting estimator with 49152 features.
Fitting estimator with 43691 features.
Fitting estimator with 38230 features.
Fitting estimator with 32769 features.
Fitting estimator with 27308 features.
Fitting estimator with 21847 features.
Fitting estimator with 16386 features.
Fitting estimator with 10925 features.
Fitting estimator with 5464 features.
Fitting estimator with 3 features.
```
This doesn't seem to be consistent with the integer step specification (also in general I don't think users would want percentage step to behave in this way).  The reason I would like percentage step to work as mentioned above is that there are use cases where one starts with lot of features and would like to fine-tune each RFECV iteration and remove fewer features as it gets closer to zero.

user: hermidalc
issue_10355: state: open
title: VotingClassifier.predict_proba with voting='hard'
body: At the moment, VotingClassifier cannot predict_proba with hard voting. I would do it the following way: 

![codecogseqn 4](https://user-images.githubusercontent.com/13658554/34314903-4a987832-e779-11e7-98a6-1e7e079f0864.gif)

Where P(y_i=c|x_i) - Probability of outcome y_i
C_n(x_i) - Result of classifier n for sample x_i in [0, 1]
Classifiers 1..n..N (Voters)
and Samples 1..i..I

In the multi-label case, it would raise an Exception in the first version.
If I create a PR, would it be accepted?
user: saddy001
issue_10329: state: open
title: NMF n_components are not properly getting reflected in output when using Grid Search CV
body: When I executed the example code from this [link ](http://scikit-learn.org/0.18/auto_examples/plot_compare_reduction.html
) 
and analyzed the grid search output (grid.cv_results_['params']), n_components are not properly getting reflected in output.

Posting a small snippet of output of grid.cv_results_['params']:
```
{'classify__C': 1000,
  'reduce_dim': NMF(alpha=0.0, beta_loss='frobenius', init=None, l1_ratio=0.0, max_iter=200,
    **n_components=None,** random_state=None, shuffle=False, solver='cd',
    tol=0.0001, verbose=0),
  'reduce_dim__n_components': 2},
 {'classify__C': 1000,
  'reduce_dim': NMF(alpha=0.0, beta_loss='frobenius', init=None, l1_ratio=0.0, max_iter=200,
    **n_components=None**, random_state=None, shuffle=False, solver='cd',
    tol=0.0001, verbose=0),
  'reduce_dim__n_components': 4},
 {'classify__C': 1000,
  'reduce_dim': NMF(alpha=0.0, beta_loss='frobenius', init=None, l1_ratio=0.0, max_iter=200,
    **n_components=None**, random_state=None, shuffle=False, solver='cd',
    tol=0.0001, verbose=0),
  'reduce_dim__n_components': 8},
```
where reduce_dim__n_components are updating for NMF but not the actual n_components in NMF

Thanks,
Pat


user: patmonsoon
issue_10323: state: open
title: [WIP] : Added assert_consistent_docs() and related tests
body: 
Fixes #9388 

Added a function to check for consistency between docstring of objects.

In this approach there is a python dictionary for each of Parameters, Attributes and Returns. Indexed by the name of the parameter/attribute/return with the value being a Python dictionary containing its type definition and description. The function checks for each object if the docstring is identical (excluding whitespaces) for a given parameter/attribute/return in the main dictionary. If a new parameter/attribute/return are found they are added in the main dictionary.
user: amanp10
issue_10315: state: open
title: Need contribute docs on what to consider when building a meta-estimator
body: Things like:
* clone
* Deep get_params works automatically in most cases
* Presence of _estimator_type
* Presence of prediction methods
* Duck typing of methods after fitting
* Parameter setting and getting overrides
* Dealing with pre-fitted estimators?

user: jnothman
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_10289: state: open
title: Generic benchmarking/profiling tool
body: We have not been proficient at documenting the estimated runtime or space complexity of our estimators and algorithms. Even were we to document asymptotic complexity functions, it would not give a realistic estimate for all parameter settings, etc. for a particular kind of data. Rather we could assist users in estimating complexity functions empirically.

I would like to see a function something like the following:

```python
def benchmark_estimator_cost(est, X, y=None, fit_params=None,
                             vary_n_samples=True, vary_n_features=False,
                             n_fits=100, time_budget=300, profile_memory=True):
    """Profiles the cost of fitting est on samples of different size

    Parameters
    ----------
    est : estimator
    X : array-like
    y : array-like, optional
    fit_params : dict, optional
    vary_n_samples : bool, default=True
        Whether to benchmark for various random sample sizes.
    vary_n_features : bool, default=False
        Whether to benchmark for various random feature set sizes.
    n_fits : int, default=100
        Maximum number of fits to make while benchmarking.
    time_budget : int, default=300
        Maximum number of seconds to use overall.  Current fit will
        be stopped if the budget is exceeded.
    profile_memory : bool, default=True
        Whether to include memory (or just time) profiling. Memory
        profiling will slow down fitting, and hence make fit_time
        estimates more approximate.

    Returns
    -------
    results : dict
        The following keys are each mapped to an array:

        n_samples
            The number of samples
        n_features
            The number of samples
        fit_time
            In seconds
        peak_memory
            The memory used at peak of fitting, in KiB.
        model_memory
            The memory in use at the end of fitting, minus that at the
            beginning, in KiB.

    models : dict
        keys 'peak_memory', 'model_memory' and 'fit_time' map to polynomial
        GP regressors whose input is n_samples and n_features and whose
        outputs are each of those targets.

    errors : list of dicts
        lists the parameters that resulted in exceptions
    """
```

This would run `fit` successively for different values of `n_samples` (logarithmically spaced, perhaps guided by a gaussian process) to estimate the function for fitting complexity, within budget. I have not thought extensively about exactly what sampling strategy would be followed. If this is implemented for the library, we would consider experimental and the algorithm subject to change for a little while.

What do others think?
user: jnothman
issue_10266: state: open
title: added sammon's mapping dimensionality reduction in manifold
body: [https://en.wikipedia.org/wiki/Sammon_mapping]

This is my first time contributing to `sklearn`. Please be patient with me. 
user: Eight1911
labels: Move to scikit-learn-extra: This PR should be moved to the scikit-learn-extras repository
issue_10236: state: open
title: solver 'lbfgs' in MLPRegressor doesn't print debug info when verbose=True
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
solver 'lbfgs' in MLPRegressor doesn't print debug info when verbose=True

#### Steps/Code to Reproduce

```python
from sklearn.neural_network import MLPRegressor
import numpy as np
x = np.random.random((100, 10))
y = np.random.random((100, 5))

solvers = ['lbfgs', 'sgd', 'adam']

for model in solvers:
    reg = MLPRegressor(solver=model, verbose=True, max_iter=10)
    print (' fitting ', model)
    reg.fit(x, y)
```

<!--
Example:
--
#### Expected Results
No verbose is printed for lbfgs.
<!-- Example: No error is thrown. Please paste or describe the expected results.-->


#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
 fitting  lbfgs
 fitting  sgd
Iteration 1, loss = 0.20245204
Iteration 2, loss = 0.19934848
Iteration 3, loss = 0.19500652
Iteration 4, loss = 0.18963213
Iteration 5, loss = 0.18342827
Iteration 6, loss = 0.17658212
Iteration 7, loss = 0.16926686
Iteration 8, loss = 0.16163198
Iteration 9, loss = 0.15381536
Iteration 10, loss = 0.14593360
 fitting  adam
Iteration 1, loss = 0.12710858
Iteration 2, loss = 0.11750231
Iteration 3, loss = 0.10845799
Iteration 4, loss = 0.09998020
Iteration 5, loss = 0.09209353
Iteration 6, loss = 0.08480344
Iteration 7, loss = 0.07810636
Iteration 8, loss = 0.07199670
Iteration 9, loss = 0.06646132
Iteration 10, loss = 0.06152093
#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
Windows-10-10.0.16299-SP0
Python 3.6.3 |Anaconda custom (64-bit)| (default, Oct 15 2017, 03:27:45) [MSC v.1900 64 bit (AMD64)]
NumPy 1.13.3
SciPy 0.19.1
Scikit-Learn 0.19.1

<!-- Thanks for contributing! -->

user: guidobiosca
issue_10233: state: open
title: Possible bug when combining SVC + class_weights='balanced' + LeaveOneOut
body: This piece of code yields perfect classification accuracy for random data:
```python
import numpy as np
from sklearn.model_selection import cross_val_score, LeaveOneOut
from sklearn.svm import SVC

scores = cross_val_score(SVC(kernel='linear', class_weight='balanced', C=1e-08), 
                         np.random.rand(79, 100), 
                         y=np.hstack((np.ones(20), np.zeros(59))), 
                         cv=LeaveOneOut())
print(scores)
```
The problem disappears when using `class_weight=None` or another CV.

Is it a bug or am I missing something?

Tested with version 0.19.1 of scikit-learn on Ubuntu Linux.
user: m-guggenmos
labels: Hard: Hard level of difficulty
issue_10186: state: open
title: Support for masked arrays in MDS
body: #### Reference Issues/PRs
None

#### What does this implement/fix? Explain your changes.
Multidimensional scaling (MDS) fails for masked arrays, producing a false embedding.  This was due to the absence of any mask handling.  This code adds support for masked arrays for MDS, so MDS can be done with distance matrices that contain masked, i.e. unknown, elements.  
user: rgerkin
labels: Needs Decision: Requires decision
issue_10164: state: open
title: Document what cost function LogisticRegression minimizes for each solver
body: #### Description

Different choices of solver for sklearn's LogisticRegression optimize different cost functions - its highly confusing behavior, particularly of concern if you want to publish what cost function you're using. In particular:

```
sklearn.LR(solver=liblinear) minimizes: L + lam*Rb
sklearn.LR(solver=others) minimizes:    L + lam*R
statsmodels.GLM(bionomial) minimizes:   L/n + lam*Rb
```
where:

```
lam = 1/C
L = logloss
n = training sample size
R = square of L2 norm of feature weights
Rb =square of L2 norm of feature weights and intercept
```

I was a little surprised to find that the logloss is not normalized by the training set size. I think this is uncommon, and means the effective C changes based on the amount of training data. Good thing, bad thing? Not sure, but it seems unusual, but more importantly, what is minimized should be explicit.

PS. https://github.com/scikit-learn/scikit-learn/pull/10001 --- excellent idea! The default liblinear cost function is just plain confusing.

#### Steps/Code to Reproduce

There's an example to show the different weights here:

https://stackoverflow.com/questions/47338695/why-does-the-choice-of-solver-result-in-different-weight-in-sklearn-logisticreg


#### Expected Results


#### Actual Results

#### Versions

1.19.1

<!-- Thanks for contributing! -->

user: stuz5000
issue_10163: state: open
title: [WIP] Fix parallel votingClassifier prediction method needed
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


Fix #7755.

Add top level methods for `predict` and `predict_proba`.

Have not been able to test it though (tests pass).
Would appreciate some help with testing.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: kameranis
issue_10158: state: open
title: Enable warnings on travis (and make sure they are reasonable)
body: We just disabled warnings on travis in #9840.
I don't think we should do that. I've been a bit absent lately, but I think the current state of the warnings is pretty bad. Many of these seem recent changes that require cleanup.

- [x] catch and ignore all `ConvergenceWarning`s raised in `sklearn.utils.estimator_checks` done in https://github.com/scikit-learn/scikit-learn/pull/11151. 
- [x] catch deprecations of min_impurity_split in tree and tree ensembles, make sure it's not used anywhere else.
- [x] catch deprecations of random_state in SVC tests, make sure it's not used anywhere else.
user: amueller
labels: Easy: Well-defined and straightforward way to resolve
issue_10148: state: open
title: Problems with score_samples in BayesianGaussianMixture
body: I have noticed that on many problems the `score_samples()` function in `sklearn.mixture.BayesianGaussianMixture` consistently gives lower test set likelihoods than just taking the posterior mean parameters from the mixture and plugging them into a regular mixture of Gaussians likelihood.

I have coded up a demo on a toy problem to illustrate:

```python
import numpy as np
import scipy.stats as ss
from scipy.special import logsumexp
from sklearn.mixture import BayesianGaussianMixture, GaussianMixture


def loglik_mixture(X, w, means, covs):
    N = X.shape[0]

    w = w / np.sum(w)  # Just to be sure normalized

    loglik = np.zeros((N, len(w)))
    for ii in range(len(w)):
        mu = means[ii, :]
        S = covs[ii, :, :]
        gauss_part = ss.multivariate_normal.logpdf(X, mu, S)
        loglik[:, ii] = np.log(w[ii]) + gauss_part
    loglik = logsumexp(loglik, axis=1)
    return loglik


def simple_data():
    x = np.random.randn(1000, 2) + 1.0
    idx = np.random.rand(1000) <= 0.5
    x[idx, :] = -1 * x[idx, :]
    return x

np.random.seed(1234)

delta = []
for _ in range(500):
    x_train = simple_data()
    x_test = simple_data()

    gmm = GaussianMixture(n_components=2, covariance_type='full')
    gmm.fit(x_train)
    loglik0 = gmm.score_samples(x_test)
    w, means, covs = gmm.weights_, gmm.means_, gmm.covariances_
    loglik1 = loglik_mixture(x_test, w, means, covs)
    # Demonstrate that loglik_mixture() is correct
    np.testing.assert_allclose(loglik0, loglik1)
    print('-' * 10)
    print(f'MLE GMM loglik {np.mean(loglik0)}')

    bgmm = BayesianGaussianMixture(n_components=2, covariance_type='full')
    bgmm.fit(x_train)
    loglik_bayes0 = bgmm.score_samples(x_test)
    w, means, covs = bgmm.weights_, bgmm.means_, bgmm.covariances_
    loglik_bayes1 = loglik_mixture(x_test, w, means, covs)
    print(f'VB GMM loglik (built-in) {np.mean(loglik_bayes0)}')
    print(f'improvement {np.mean(loglik_bayes0) - np.mean(loglik0)}')
    print(f'VB GMM loglik {np.mean(loglik_bayes1)}')
    print(f'improvement {np.mean(loglik_bayes1) - np.mean(loglik0)}')

    delta.append(np.mean(loglik_bayes1) - np.mean(loglik_bayes0))
print(np.mean(delta))
print(np.mean(np.array(delta) > 0))
```
This gives for instance:
```
MLE GMM loglik -3.362221
VB GMM loglik (built-in) -3.367131
improvement -0.004911
VB GMM loglik -3.361099
improvement 0.001121
```
The built in score for VBGMM is consistently worse than MLE GMM but often better when the posterior mean is plugged in as a point estimate in loglik_mixture().  On this problem, the loglik_mixture() likelihood is always around 0.006 nats higher than the built in score function.

This makes me wonder if there is some normalization issue in the likelihood in _estimate_log_prob().  I am not sure where the derivation for the code in lines 690-698 is.

The documentation says it implements Blei and Jordan (2006), which has the posterior predictive in eqn (23).  But that is only an approximation and not necessarily even normalized.

If score_samples is used for evaluation purposes, it is not fair to compare VB-GMM to other models if its likelihood is not even normalized!  Some more investigation is needed here.
user: rdturnermtl
labels: Bug: triage: Reported bugs that are not confirmed
issue_10141: state: open
title: Documentation error in GaussianMixture
body: In the documentation of sklearn.mixture.GaussianMixture, it says that precisions_cholesky_ is:
"The cholesky decomposition of the precision matrices of each mixture component."
However in lines 317--321 of sklearn.mixture.gaussian_mixture.py it is clearly computing the inverse of the cholesky rather than the cholesky of the inverse.  These are not the same:
```
import numpy as np

np.random.seed(0)
S = np.random.randn(5, 5)
S = np.dot(S, S.T)
IC = np.linalg.inv(np.linalg.cholesky(S).T)
CI = np.linalg.cholesky(np.linalg.inv(S)).T

print IC
print CI
```
gives
```
[[ 0.2801735   0.05324512 -0.24035224  0.15420992  0.49081071]
 [ 0.          0.70922139 -0.67751324  0.23295284 -6.89597681]
 [ 0.          0.          0.89988635 -0.67437444 -3.85334538]
 [ 0.          0.          0.          0.72994649  6.24915248]
 [ 0.          0.          0.          0.          3.76407547]]
[[ 0.63543472 -4.9542323  -3.48037026  5.00400095  2.90737741]
 [ 0.          4.90166957  1.74700377 -3.69934856 -2.35698345]
 [ 0.          0.          0.97357464 -0.71268093 -0.27514477]
 [ 0.          0.          0.          0.5929664   0.09843469]
 [ 0.          0.          0.          0.          0.27323201]]
```
I think the code is correct, but the documentation needs to be corrected.  The same issue applies to BayesianGaussianMixture.
user: rdturnermtl
issue_10122: state: open
title: GaussianMixture default 'tol' parameter too large?
body: While working with a synthetic dataset, I have noticed that using the default tol=1e-03 the final "weights" of the Gaussian Mixture components is "significantly" off from the true values. Following is the data generation process and the fitted model parameters for a couple of 'tol' values:

```
from __future__ import division, print_function
import numpy as np
from scipy.stats import multivariate_normal
from sklearn.mixture import GaussianMixture

# Generate data
np.random.seed(1234)
n_samples_X1 = 300
n_samples_X2 = 100
means_true = {0: [2, 4], 1: [0, 0]}
covar_true = {0: [[5, 2],
                  [2, 6]],
              1: [[4, 2],
                  [2, 3]]}
X1 = multivariate_normal(means_true[0], covar_true[0]).rvs(
    size=n_samples_X1)
X2 = multivariate_normal(means_true[1], covar_true[1]).rvs(
    size=n_samples_X2)
X = np.concatenate((X1, X2), axis=0)

# Fit and print results
tol = [1e-3, 1e-6]
for t in tol:
    gmm = GaussianMixture(n_components=2, tol=t)
    gmm.fit(X)
    print("Iteration: ", gmm.n_iter_)
    print("Means:\n", gmm.means_,
          "\n\nCovariance Matrix:\n", gmm.covariances_,
          "\n\nWeights: \n", gmm.weights_,
          "\n\nLog Likelihood: \n", gmm.lower_bound_)
```
The 300/100 sample divide for X1 and X2 should mean that the final weight on X1 should converge to 0.75 while for X2 should converge to 0.25. But using the default tol the result is as follows:

```
-----------
 tol=1e-03
-----------
Iteration:  7
Means:
 [[ 0.36914419  0.69188067]
 [ 2.13114152  4.32671517]] 

Covariance Matrix:
 [[[ 3.95984165  2.40375214]
  [ 2.40375214  4.83616171]]

 [[ 4.26702227  1.04679082]
  [ 1.04679082  4.79127014]]] 

Weights: 
 [ 0.40842607  0.59157393] 

Log Likelihood: 
 -4.51294936356
```
In other words, the final weight for the two components is about 40-60 when it should have been closer to 25-75. But if the value is set to tol=1e-06, the final outcome is as follows:
```
tol=1e-06
-----------
Iteration:  194
Means:
 [[ 0.02089568 -0.15599767]
 [ 1.88419257  3.86129407]] 

Covariance Matrix:
 [[[ 3.94321677  2.24828336]
  [ 2.24828336  3.60910343]]

 [[ 4.33331742  1.55546864]
  [ 1.55546864  5.4008179 ]]] 

Weights: 
 [ 0.25368834  0.74631166] 

Log Likelihood: 
 -4.50970105471
```
...which is the result we expected.

I realize that this is only one example but I figured it was sufficiently non-trivial to at least post here for discussion. 

PS: For what it's worth (and though not directly comparable), a point of reference might be [Scipy solvers](https://docs.scipy.org/doc/scipy/reference/optimize.minimize-bfgs.html) that seem to use gtol=1e-05.

Test Environment:
```
Python 3.5.2
NumPy 1.13.3
SciPy 1.0.0
Scikit-Learn 0.19.1
```
user: ashimb9
issue_10121: state: open
title: [MRG] LoOP: Local Outlier Probabilities
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs

#### What does this implement/fix? Explain your changes.
This branch implements LoOP (Local Outlier Probabilities), an unsupervised outlier detection approach that's similar to LOF (Local Outlier Factor) but that normalizes scores in the range [0, 1]. 

#### Any other comments?
I'm looking for contributing authors / developers interested in contributing and adding their names to the list of authors by completing this feature. Due to time constraints, I am unable to perform any rigorous testing and am looking for collaborators to add any missing / additional features and ensure proper functionality. 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: vc1492a
labels: Move to scikit-learn-extra: This PR should be moved to the scikit-learn-extras repository
issue_10119: state: open
title: Improve test for MDS parallel processing case - requires update in MDS
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
There is a test that MDS does not create an error when performing parallel processing (`n_jobs > 1`)
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/manifold/tests/test_mds.py#L55-L61

That test should also assert that the result is comparable to with `n_jobs = 1`.  

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->
Proposed test:
````python
def test_MDS_parallel():
    sim = np.array([[0, 5, 3, 4],
                    [5, 0, 2, 2],
                    [3, 2, 0, 1],
                    [4, 2, 1, 0]])
    mds_clf = mds.MDS(metric=False, n_jobs=1, n_init=4, dissimilarity="precomputed")
    result1 = mds_clf.fit(sim)

    # TODO: if sim is modified my MDS.fit then set it back before the next test

    mds_clf = mds.MDS(metric=False, n_jobs=4, n_init=4, dissimilarity="precomputed")
    result2 = mds_clf.fit(sim)

    assert_array_almost_equal(result1, result2, decimal=3)
````
... as a replacement for
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/manifold/tests/test_mds.py#L55-L61

The smacof algorithm involves an iterative search from `n_init` random initial solutions. In the parallel case each job searches from a new defined seed.  In the non-parallel case, `random_state` is used to continue searching from a single starting seed.  These varying approaches don't produce the same results.  The non-parallel case must be modified to search from increasing seeds in order to be consistent with the parallel case.  That will allow for consistent output when switching between the two parameter options and also allow for this new test to pass.

See
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/manifold/mds.py#L248-L271

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->


<!-- Thanks for contributing! -->

user: jsoutherland
labels: module:test-suite: everything related to our tests
issue_10108: state: open
title: Feature Request: KDE for Bounded Data
body: **Feature request:**
It would be great to have a KDE that worked on bounded data (i.e. positive-definite data). There are many flavors of this, including implementations in the python world (i.e. [PyQt-Fit](https://pypi.python.org/pypi/PyQt-Fit/1.1.5)), and there are other approaches in other packages (i.e. [log-spline density in R](https://cran.r-project.org/web/packages/logspline/index.html)). Any reasonable implementation would be appropriate.

Stack Overflow has numerous examples of people trying to find a way to implement this cleanly. For example, [this example](https://stackoverflow.com/questions/41894731/boundary-correction-for-multivariate-kde-in-python), [this example](https://stackoverflow.com/questions/29083429/python-scipy-kde-fit-scaling), or [this example](https://stackoverflow.com/questions/21809876/lower-bound-to-kernel-density-estimation-with-seaborn-for-matplotlib-in-python#21815194)
user: jdossgollin
issue_10082: state: open
title: No complete list of necessary checks for creating new classes in the docs (or anywhere else).
body: @amueller asked me to open this issue.

When creating new estimators using any of the base classes, or inheriting from any of the main sklearn classes, there is no complete list (or any list?) of all of the checks that need to be run through to create a well-formed estimator or sklearn-compatible class. check_array covers a lot of things, but definitely nowhere near all things.

This is important, because the learning curve for contributing new classes and methods to sklearn is unnecessarily steep, and it would be much easier to contribute if there was, in the docs, a list of all the necessary checks that need to be run through for different types of estimators and sklearn classes, and a clear explanation of how and when the different checking utils should be used.


user: voxmenthe
issue_10068: state: open
title: Issues encountered with the memory option in Pipeline
body: I'm very interested in using the `memory` option in `Pipeline` so I can organise my code in a simple fashion. However I've found that it does not scale well or there seem to be some caveats one could not expect:

- is the input train data is too big (~several GB), joblib.memory takes a very long time to hash it, it can considerably slow the execution in an unexpected way.

- the documentation hints that `Caching the transformers is advantageous when fitting is time consuming.`. However, the `fit_transform` is cached, so not only the fitted transformer but also the transformed data of the train data seems to be cached ? Then, it is also advantageous when transforming is time consuming, however this can quickly add up to a considerable space taken on the hard drive.

- finally, it the code of a transformer change, but neither his methods nor his attributes, it seems to me that the hash will not change (because the code of _fit_transform_one does not). That's something the user could be warned about (need to wipe the cache when the code of a currently cached transformer has been altered) or it can happen to load from the cache the previous version of a transformer by mistake.

user: fcharras
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_10043: state: open
title: [WIP] Voting class logit vote
body: #### Reference Issues/PRs
Fixes #9933

#### What does this implement/fix? Explain your changes.
It adds the possibility to pick odd-based voting rule to the `VotingClassifier`. WIP

#### Any other comments?
See [this paper](https://arxiv.org/pdf/1302.0540.pdf) for why this might be a good voting rule (based on a game-theoretical aproach)

user: CoderPat
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_10021: state: open
title: Common test for inverse_transform
body: Add a common test that ``inverse_transform`` if present does all the standard input validation (sparse data, missing values, dtype). We might want to have some checks on ``est.inverse_transform(est.transform())``. I'm not sure what. Definitely shape. Is it reasonable to expect that ``est.transform(est.inverse_transform())`` is the identity?
Maybe we could do
```python
X_t = est.transform(X)
```
and then check
```python
assert_almost_equal(X_t, est.transform(est.inverse_transform(X_t))
```
But I'm not sure that makes sense for anything but PCA?
user: amueller
labels: Needs Decision: Requires decision
module:test-suite: everything related to our tests
issue_10010: state: open
title: Different(wrong?) meaning of pos_label=None
body: Currently, in scikit-learn, we have multiple definitions of pos_label=None and the doc seems not clear, which might confuse users.
(1) precision/recall/f1/fbeta: do not support pos_label=None
(2) brier_score_loss: default pos_label=None, means the greater of two values (the implementation currently under fix by me in #9562)
(3) precision_recall_curve/roc_auc_curve: default pos_label=None, means pos_label=1

According to @jnothman 
 
> Actually, I suspect None should mean "the greater of two values" (unless there is only one value present).

possible solutions:
(1)deprecate pos_label=None and change the default value to 1
(2)correct the behaviour of pos_label=None in precision_recall_curve/roc_auc_curve
(3)ignore the difference and simply update the doc

Personally, I advocate for (1), at least in precision_recall_curve/roc_auc_curve, because
(1) It will not influence user's code too much, unless you are using something like y_true = ['a', 'b'] and don't provide a pos_label. In this case, it seems not reasonable to let scikit-learn set pos_label='b' for you. 
(2) We don't have a correct implementation of pos_label=None in scikit-learn currently. The meaning of pos_label=None is hard to understand by user and hard to define. If there are two labels, ok, we pick the greater one. How about the situation if there is only one label? Pick that one? I'm afraid it is not always the case ,e.g., when there's only negative samples.

cc @jnothman
user: qinhanmin2014
issue_10003: state: open
title: Add sklearn.metrics.cumulative_gain_curve and sklearn.metrics.lift_curve
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description

I recently added `plot_cumulative_gain` and `plot_lift_curve` methods to https://github.com/reiinakano/scikit-plot. To do this, I built an adhoc version of `cumulative_gain_curve` closely following the `sklearn.metrics.roc_curve` interface at https://github.com/reiinakano/scikit-plot/blob/master/scikitplot/helpers.py#L157. Let me know if `sklearn.metrics.cumulative_gain_curve` is something you'd be interested in adding into scikit-learn. I could add example docs for plotting gain and lift curves as well. 

Reference I followed for lift and gain: https://www.ibm.com/support/knowledgecenter/en/SSLVMB_23.0.0/spss/tutorials/mlp_bankloan_outputtype_02.html

![plot_cumulative_gain](https://user-images.githubusercontent.com/18363734/32030142-940d87f8-ba2b-11e7-8280-f6e4fb782cf7.png)

![plot_lift_curve](https://user-images.githubusercontent.com/18363734/32030157-b64672c6-ba2b-11e7-9d59-ed5cce68fde2.png)


user: reiinakano
issue_9953: state: open
title: Scatter Separability Criterion (SSC) for clustering algorithms
body: I don't think I've seen references to the _Scatter Separability Criterion_ (SSC) metric anywhere yet - neither in the Issues section here, nor on the mailing list.

Has it been ever considered and if yes, what are the reasons it hasn't made its way into the _sklearn_ module yet?
Would a potential PR be encouraged?

**References:**
- Dy, J. G., and Brodley, C. E. (2004), "Feature selection for unsupervised learning," _Journal of Machine
Learning Research_, 5, 845-889. [[Google Scholar link]](http://www.jmlr.org/papers/volume5/dy04a/dy04a.pdf)
- _Separability Criteria for Feature Selection_ [[link]](http://fourier.eng.hmc.edu/e161/lectures/classification/node4.html)

P.S.: Apologies if I've missed relevant discussions and/or implementations which have already addressed the same question.
user: hristog
issue_9933: state: open
title: Votingclassifier uses arithmetic mean. Should be taking a mean of log odds.
body: #### Description
I notice that [VotingClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html#sklearn.ensemble.VotingClassifier 
) with voting='soft' takes averages of the predicted probabilities. But the way it should work is by taking the mean of the log odds (logits).

Think of it as a Bayesian... If I think the probability is 1 (infinite evidence in favor) and you think it's 1/2 (no evidence), combined, we think it's 1, not 3/4. If I observe something that weighs 4:1 toward hypothesis A, and you and your friend each have independent observations that weigh 2:1 toward hypothesis B, then on balance, we haven't changed our probabilities from a prior of 1/2.

So we should be taking a scipy.special.logit, then a np.mean, then a scipy.special.expit in predict_proba. Feel free to correct me.

#### Lines to be fixed:
[avg = np.average(self._collect_probas(X), axis=0,
                         weights=self._weights_not_none)](https://github.com/scikit-learn/scikit-learn/blob/ef5cb84a/sklearn/ensemble/voting_classifier.py#L241)
user: RyanCarey
issue_9900: state: open
title: New feature: correlation matrix generation
body: I was thinking it might be a good idea to enrich the positive-definite matrix generation part, by adding correlation matrix generation.

**Interest of covariance matrices**: Covariance and correlations matrices are cool because from them we can generate multi-variate data sampled from specific laws (example, mutltivariate gaussian)

 **Why correlation matrices**:  Now we can generate covariance matrices (that are symmetric positive definite matrices) from `sklearn.datasets.make_spd_matrix`. However, covariance coefficients are dependent on the variable's variances, and do not give a clear, intuitive information about variables dependencies. Correlation matrices inform us about this intrinsic dependency through scaled coefficients.

**The feature**: I think that would be cool to say for example: I want a 4x4 correlation matrix with c<sub>12</sub> low, c<sub>23</sub> high and c<sub>34</sub> high. More generally, we can even replace "low" and "high" with any value between -1 and 1;  the algorithm will try to match them (but within the complex space of correlation matrices)

**How**: Of course, generating a correlation matrix C is not as simple as simply filling a matrix with 1's on the diagonal and c<sub>ij</sub><1. Most of them won't work. Methods exist on the literature (-> [Onion Method Generation](https://www.sciencedirect.com/science/article/pii/S0047259X09000876), sorry for non-academic wifi users...) for uniform and/or more complex sampling behaviors and it may be a good idea to implement one of them.



If this generates any interest, we can discuss it together, and I am fine with implementing a batch of functions to generate such objects.

Let me know guys!

user: pierreglaser
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_9894: state: open
title: Adding a `prefit` parameter to the VotingClassifier
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
<!-- Example: Fixes #1234 -->

We don't have an issue for that but discussed this via the mailing list (https://mail.python.org/pipermail/scikit-learn/2017-October/001960.html)


#### What does this implement/fix? Explain your changes.

Adds a `prefit` parameter to the `VotingClassifier` so that it accepts prefitted estimators; currently, it's only possible to use the `VotingClassifier` upon refitting clones of the original estimators by running `fit`.


#### Any other comments?

This is an early PR for feedback. 

Below are todos to get to when the implementation details are sorted out.


- [ ] Add note to "What's New" docs
- [ ] Add appropriate unit tests for `predict`, `predict_proba`, and `transform` when `prefit=True`

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->
user: rasbt
issue_9888: state: open
title: [MRG+1] Enforce euclidean metric on LLE
body: #### Reference Issue
Fixes #9868

#### What does this implement/fix? Explain your changes.
The LocallyLinearEmbedding class uses the NearestNeighbors class to calculate the K nearest neighbors yet only allowed the use of the euclidean distance metric to calculate those. I added the arguments needed to the class (and all functions used by it) in order to allow any valid metric for nearest neighbors to be used. Also added a test to ensure that another metric can be used.

#### Any other comments?
I'm not entirely sure of the utility of changing the metric from the euclidean to another one since the error function is calculated based on the euclidean metric. Maybe I also need to change the error calculation function to be dependent on the metric? Either way it is at least possible to choose the neighbors for the embeding based the user choice of metric.
Apreciate any feedback since its my first contribution
user: CoderPat
issue_9868: state: open
title:  scikit-learn/sklearn/manifold/locally_linear.py could have a parameter for metric type
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
Local Linear Embedding uses distance metrics in the calculation of nearest neigbhors.  it uses the NearestNeighbors class to do that.  There doesnt seem to be a way to pass in the type of distance metric used.  It uses a default of euclidean (or minkowski 2).  It would be nice to pass in as a parameter different metrics to try, such as jaccard for categorical or whatever one would like.  

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->


<!-- Thanks for contributing! -->

user: aarondbaron
issue_9854: state: open
title: A Policy Proposal on Missing Data for Future Contributions
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->
As you might know, a number of us are working towards bringing various imputation transformers into Scikit-Learn. In addition to imputation transformers, there are also a number of PRs working to implement direct model estimation in the presence of missing data without explicit imputation as a preprocessing step. In other words, we are slowly moving in the direction of comprehensively supporting preprocessing and analysis based on datasets with missing values. Given this context, I just wanted to start a discussion on whether it would make sense for Scikit Learn to consider making missing-data support either a "requirement" or a "highly recommended" component in all future algorithms that will be added to the library (to the extent that such support makes sense and is established in the literature, of course). In my humble opinion, not only would this approach lead to a more "robust" set of tools but also saves a lot of time and resources compared to refactoring the code at a later time, especially in situations when it is done by somebody other than the original author themselves. I would very much appreciate it if the core developers and other contributors could comment on this issue. Thank you for your time and consideration.

TL;DR: Would it make sense for Scikit Learn to have a default *policy* of missing value support in all applicable algorithms that are added to the library in the future?
user: ashimb9
issue_9834: state: open
title: [MRG] Extending MDS to new data
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
<!-- Example: Fixes #1234 -->
Continuation of #6222.

#### What does this implement/fix? Explain your changes.
Extended MDS object to include a transform method for out-of-sample points as described here:
http://papers.nips.cc/paper/2461-out-of-sample-extensions-for-lle-isomap-mds-eigenmaps-and-spectral-clustering.pdf

SMACOF algorithm is used in the non-extendible (default) case but not for the extendible case, which requires eigen-decomposition of (dis)similarity matrix.

Originally implemented by @webdrone in #6222.   It has since been modified to alter the `fit`/`transform` API and speed up subsequent `MDS.transform()` calls

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: jsoutherland
issue_9824: state: open
title: Fitting ExpSineSquared kernel to large dimension data throws a ValueError on MacOS
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
ValueError: "array must not contain infs or NaNs" thrown when fitting an ExpSineSquared kernel to large dimension data

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->
```py
import sklearn.gaussian_process as gp
from sklearn.gaussian_process.kernels import ExpSineSquared

while True:
    xtr = np.random.rand(25,25)
    ytr = np.random.rand(25)
    model = gp.GaussianProcessRegressor(kernel=ExpSineSquared(),
                                                alpha=1e-5,
                                                n_restarts_optimizer=10,
                                                normalize_y=True)
    model.fit(xtr, ytr)
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
No error is thrown

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->

```
  File "<ipython-input-39-119d71c23c9f>", line 1, in <module>
    runfile('/Users/eric/Google Drive/Spatial conquest/Neural Network/GitHub Repository/tensorlayer/benchmarks/bug.py', wdir='/Users/eric/Google Drive/Spatial conquest/Neural Network/GitHub Repository/tensorlayer/benchmarks')

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 710, in runfile
    execfile(filename, namespace)

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 101, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/Users/eric/Google Drive/Spatial conquest/Neural Network/GitHub Repository/tensorlayer/benchmarks/bug.py", line 19, in <module>
    model.fit(xtr, ytr)

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/sklearn/gaussian_process/gpr.py", line 232, in fit
    bounds))

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/sklearn/gaussian_process/gpr.py", line 455, in _constrained_optimization
    if convergence_dict["warnflag"] != 0:

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py", line 193, in fmin_l_bfgs_b
    **opts)

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py", line 328, in _minimize_lbfgsb
    f, g = func_and_grad(x)

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py", line 278, in func_and_grad
    f = fun(x, *args)

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/scipy/optimize/optimize.py", line 292, in function_wrapper
    return function(*(wrapper_args + args))

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/scipy/optimize/optimize.py", line 63, in __call__
    fg = self.fun(x, *args)

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/sklearn/gaussian_process/gpr.py", line 209, in obj_func
    theta, eval_gradient=True)

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/sklearn/gaussian_process/gpr.py", line 419, in log_marginal_likelihood
    except np.linalg.LinAlgError:

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/scipy/linalg/decomp_cholesky.py", line 81, in cholesky
    check_finite=check_finite)

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/scipy/linalg/decomp_cholesky.py", line 20, in _cholesky
    a1 = asarray_chkfinite(a)

  File "/Users/eric/Applications/anaconda/envs/NN_on_GPU/lib/python3.6/site-packages/numpy/lib/function_base.py", line 1215, in asarray_chkfinite
    "array must not contain infs or NaNs")

ValueError: array must not contain infs or NaNs
```

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
Darwin-16.7.0-x86_64-i386-64bit
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:14:59) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
NumPy 1.13.1
SciPy 0.19.1
Scikit-Learn 0.19.0

#### Additional comment
This problem seems MacOS specific as it does not occur on my linux system with exactly the same versions
Linux-4.10.0-33-generic-x86_64-with-debian-stretch-sid
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
NumPy 1.13.1
SciPy 0.19.1
Scikit-Learn 0.19.0

In terms of investigation, the furthest i got was to see that `kernel(self.X_train_)` in sklearn/gaussian_process/gpr.py", line 419, log_marginal_likelihood was diverging to infinity

<!-- Thanks for contributing! -->

user: Ericvulpi
issue_9822: state: open
title: [MRG] Transposing clustering and classification comparisons
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
<!-- Example: Fixes #1234 -->
Fixes #9739

#### What does this implement/fix? Explain your changes.
Transposes the comparison of clustering and classification algorithms.

#### Any other comments?
Couldn't see any clever way to invert the for loops in the clustering example to do a trivial transpose, so I used a known conversion between matrix indexes to 1-dim array.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: syonekura
labels: Needs Decision: Requires decision
issue_9819: state: open
title: Standard attribute for model completion status
body: At the moment, there is no way to know why a model stopped fitting (aside from exceptions). It may be because convergence was reached (below a tolerance), or because a hard limit on iterations was hit, or possibly some other condition was met. It would be excellent if Scikit-learn models had some kind of flat attached after fitting. Perhaps `model.fit_status_` or similar. This could then be reported during usage.

For example, I'm hitting [a problem with MiniBatchKMeans clustering](https://stats.stackexchange.com/questions/304404/dealing-with-unassigned-clusters-in-minibatchkmeans), where some clusters are not assigned. The only think I know is that the `max_iter` was not met, but I don't know if it's due to the `max_no_improvement` flag, because the `tol` is too high (not in my case). It would be great if the model reported this in a standard way.
user: naught101
labels: Needs Decision - API: Requires decision regarding API
issue_9779: state: open
title: Feature request: include mean absolute error (MAE) as scorer in sklearn.linear_model.ElasticNetCV
body: <!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->

In scikit-learn v0.19.0, ElasticNet minimizes the objective function:

> `1 / (2 * n_samples) * ||y - Xw||^2_2`
> `+ alpha * l1_ratio * ||w||_1`
> `+ 0.5 * alpha * (1 - l1_ratio) * ||w||^2_2`.

In words, the objective function is mean squared error (MSE) loss on the predicted variable, plus regularization terms.  My feature request is to include an argument (with 'mse' as its default value) that would allow the user to switch to mean absolute error (MAE) loss on the predicted variable.  For the equivalent in R, see https://cran.r-project.org/web/packages/glmnet/glmnet.pdf, particularly the documentation for cv.glmnet:

> type.measure="mae" (mean absolute error) can be used by all models except ... "cox"

It would be great to also allow this functionality in sklearn.

Edit:  as @agramfort pointed out below, glmnet only supports MAE for cross-validation, but not for model fitting.  My request is to add an option for MAE in sklearn ElasticNet model fitting.  This already exists in sklearn.ensemble.GradientBoostingRegressor (loss='quantile' and alpha=0.50), if I am reading the documentation correctly.  There's also a linear-regression-with-MAE-loss option in sklearn.linear_model.HuberRegressor.html, if epsilon=0.0 ("The Huber Regressor optimizes the squared loss for the samples where `|(y - X'w) / sigma| < epsilon` and the absolute loss for the samples where `|(y - X'w) / sigma| > epsilon`").
user: atorch
issue_9763: state: open
title: warm_start for GridSearchCV and RandomizedSearchCV
body: In practice it is useful to search over some parameter space and then continue the search over some related space. We could provide a `warm_start` parameter to make it easy to accumulate results for further candidates into `cv_results_` (without reevaluating parameter combinations that have already tested).

Issues:
* What do we do when the parameter space is reduced rather than expanded; do we delete former results?
* I'm not sure that this is identical semantics to other uses of `warm_start`. Do we let things change other than the parameter space? The set of scorers for instance?
* Given that calling split multiple times on a cv splitter does not guarantee identical splits, this means we would need to store the first cv split indices and reuse them in subsequent warm started fits. This in turn means that we cannot use warm start in a way that allows us to change the CV splitter (which I think @ogrisel mentioned as a potential use of grid search a while ago).
* Does calling `RandomizedSearchCV(n_iter=10, warm_start=True)`'s fit twice mean that it should do 10 iterations for each call, or 10 in total (0 in the second call)?
* Is this unnecessary complexity to add to GridSearchCV when users can just concatenate the multiple `cv_results_` dataframes?
user: jnothman
issue_9758: state: open
title: MAINT remove unused stop parameter from compute_gradient
body: Ping @tommoral, @ogrisel 
user: jnothman
labels: Needs Decision: Requires decision
issue_9756: state: open
title: [request] Reference for Agglomerative Clustering
body: It could be helpful to have a base paper referencing the implementation adopted at the AgglomerativeClustering class.

Does anyone have such reference, so that we could attach to the documentation?

user: LPugens
issue_9741: state: open
title: [MRG] run check_estimator on meta-estimators
body: Fixes #9443.

After #9716 I think it's time to add these checks.
user: amueller
issue_9739: state: open
title: Transpose the clustering and classification comparisons
body: The images at http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html and http://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html are getting very wide as we add more clusterers and classifiers.

We add datasets more slowly, so these images should be transposed: algorithms by row and datasets by column.
user: jnothman
labels: Needs Decision: Requires decision
issue_9722: state: open
title: Bad results for LabelPropagation (and LabelSpreading) when using any kernel other than "rbf" or "knn"
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
Poor results always obtained when using sklearn.semi_supervised.LabelPropagation (or LabelSpreading) with any (callable) kernel other than the predefined "knn", "rbf", or sklearn.metrics.pairwise.rbf_kernel

#### Steps/Code to Reproduce
Example:
```python
from sklearn.semi_supervised import LabelPropagation, LabelSpreading
from sklearn.metrics.pairwise import pairwise_kernels, rbf_kernel

def my_kernel(X, Z):
        # This works good :
	return rbf_kernel(X, Z)
        # This seems to always work bad for any choice of metric (and parameters), except for "rbf" :
	return pairwise_kernels(X, Z, metric="cosine")

h = LabelSpreading(kernel=my_kernel, n_neighbors=7, alpha=0.2).fit(X_labelled+X_unlabelled, Y_labelled+[-1 for _ in X_unlabelled])
print "Result:", 100*accuracy_score( Y_test, h.predict(X_test) )
```

Here is a complete other example using the optical digits dataset:
```python
from sklearn.metrics import accuracy_score
from sklearn import datasets 
from sklearn.semi_supervised import LabelPropagation, LabelSpreading
from sklearn.utils import shuffle
from sklearn.metrics.pairwise import pairwise_kernels, rbf_kernel

def my_kernel(X, Z):
	# return rbf_kernel(X, Z, gamma=0.1) # works good
	return pairwise_kernels(X, Z, metric="cosine")  # works bad
	
if __name__ == "__main__":
	digits = datasets.load_digits()
	X, Y = shuffle( list(digits.data), list(digits.target) )
	Lx = X[:50]; Ly = Y[:50]; Ux = X[50:]; Uy = Y[50:]
	
	print Ly
	h = LabelSpreading(kernel=my_kernel, n_neighbors=7, alpha=0.2).fit(Lx+Ux, Ly+[-1 for _ in Ux])
	print "Result:", 100*accuracy_score( Uy, h.predict(Ux) )
```

#### Expected Results
I expected kernels such as: sigmoid, polynomial, poly, linear, cosine (tested with various parameters) to perform reasonably good, but all of those give extremely poor results.

#### Actual Results
Only kernel = "knn", or "rbf" (or a callable corresponding to an rbf kernel) gives good results.

#### Versions
Linux-3.10.0-514.6.1.el7.x86_64-x86_64-with-centos-7.3.1611-Core
('Python', '2.7.5 (default, Nov  6 2016, 00:28:07) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]')
('NumPy', '1.11.2')
('SciPy', '0.18.1')
('Scikit-Learn', '0.19.0')


<!-- Thanks for contributing! -->

user: HTCode
issue_9698: state: open
title: Use a safe and short repr in error messages and warning
body: We print the value of an offending parameter in many of our error messages and warnings. However, when the object failing printing, or when its representation is too long, the resulting message is not useful.

We should:

- Use a safe_repr function to never fail printing (I am pasting an example below, with a test)
- Only print the 300 first characters (something like this) of it's return, using a "short_repr'

These two functions should be added in the utils submodule and used in error messages and warnings in the codebase.

```py
def safe_repr(value):
    """Hopefully pretty robust repr equivalent."""
    # this is pretty horrible but should always return *something*
    try:
        return pydoc.text.repr(value)
    except KeyboardInterrupt:
        raise
    except:
        try:
            return repr(value)
        except KeyboardInterrupt:
            raise
        except:
            try:
                # all still in an except block so we catch
                # getattr raising
                name = getattr(value, '__name__', None)
                if name:
                    # ick, recursion
                    return safe_repr(name)
                klass = getattr(value, '__class__', None)
                if klass:
                    return '%s instance' % safe_repr(klass)
            except KeyboardInterrupt:
                raise
            except:
                return 'UNRECOVERABLE REPR FAILURE'


def short_repr(obj):
     msg = safe_repr(obj)
     if len(msg) > 300:
        return msg = '%s...' % msg
     return msg

# For testing (in a test file, not in the same file)
class Vicious(object):
    def __repr__(self):
        raise ValueError


def test_safe_repr():
    safe_repr(Vicious())


```

safe_repr is borrowed from joblib, but as it is not exposed in the public API, we shouldn't import it from our vendored version of joblib (elsewhere, the "unvendoring" performed by debian will break the import)
user: GaelVaroquaux
issue_9695: state: open
title: Rationalize metric argument in T-SNE
body: The API of T-SNE have a `metric` argument to allow changing the metric in the input space. The original paper was developed using only the euclidean distance and changing the metric might change a bit the math. 

The main issue is to understand what is going on with the computation of the conditional probabilities. The original paper states that if `d_ij` is the euclidean distance between `x_i` and `x_j`, then the conditional probability is given by
``` 
p(x_j | x_i) = exp{ d_ij^2 / sig_i} / Z
```
where Z is a normalization constant and `sig_i` is a parameter, computed to fix the perplexity of the model. _If `d_ij` is not the euclidean distance, does it make sense to use this formula?_ 
The current implementation for other metric than the euclidean uses the formula without the square, so
``` 
p(x_j | x_i) = exp{ d_ij / sig_i} / Z
```
If it is fine, this should be documented. Else, shall we change it to use the square or another distribution? Another solution is to deprecate the `metric` argument.

A second point is the use of `init='pca'`. It is a very natural initialization for the euclidean metric but should be benchmarked for other to see if it gives an interesting starting point or not.

This issue is a follow-up of #9623 .
user: tomMoral
labels: Needs Decision: Requires decision
issue_9661: state: open
title: LatentDirichletAllocation: Modernization of Inference
body: Firstly, I'm rather new around here; I hope this is the correct place for proposing replacement of algorithms. If it isn't, it would be awful kind of someone to point me in the right direction.

#### Summary
LatentDirichletAllocation uses a very old (2003) algorithm to estimate its parameters. I tried using the sklearn function for my research, but found that it was unsatisfactory compared to newer algorithms. I have implemented some newer algorithms, and would be happy to get one of them in a format conforming to sklearn's requirments and contribute them. I only have experience in using batch estimation (so no online), but I'd be happy to look into newer online algorithms as well.

#### Proposed New Algorithm

--Collapsed Variational Bayes (Zeroth Order Taylor Expansion)--

Introduced in 2012 in [Asuncion et al](https://arxiv.org/pdf/1205.2662.pdf), the Zeroeth Order Taylor approximation to Collapsed Variational Bayes (often abbreviated CVB0) is a much more modern algorithm than the original variational bayes algorithm proposed in [Blei Ng Jordan, 2003](https://endymecy.gitbooks.io/spark-ml-source-analysis/content/%E8%81%9A%E7%B1%BB/LDA/docs/Latent%20Dirichlet%20Allocation.pdf). 

It is essentially a deterministic version of Collapsed Gibbs Sampling (not introduced in [Wilson and Chew, 2010](http://www.aclweb.org/anthology/N10-1070), but I find their description quite understandable), wherein the topic-word propensity matrix (called $\beta$ in Blei et al and sometimes called $\phi$ in newer articles) as well as the document-topic prevalence vector (called $\theta$ in Blei et al) are analytically integrated out. This results in a posterior only on the word-topic associations (called $z$ in Blei et al), over which a variational distribution is defined. Unfortunately, this cost function is not analytic, but taylor series approximations yield good empirical results.

#### Advantages of New Algorithm

Based on my personal experience and reviewing relevant literature, I see the benefits of the new algorithm as threefold:

1) **Considerably faster** -- Asuncion et al report a tenfold increase. I have observed much higher in practice on certain datasets (though, admittedly, I did not spend much time optimizing my cpp implementations of either algorithm).
2) **More accurate** -- Theoretically, the process of integrating out certain parameters results in a lower bound on the loglikelihood that is strictly better than the lower bound induced by traditional VB. Empirically, taylor approximations of this bound are still able to beat traditional VB.
3) **Increase in convexity of cost surface** -- This is something I have not seen in the literature, but have observed myself. It is very important in my research that running the algorithm twice should produce the same result, something which seems not to occur with traditional VB. The newer algorithm seems to be much better at avoiding multimodality of the cost surface.

\vspace{5em}

I've benefited very much over the last several years from using sklearn, both as an academic in research and classes, as well as during internships in industry. I feel an obligation to contribute, now that I can, should these improvements be desired. My implementation is in C, but I can easily create a python wrapper for it (or re-write in python, if that's desired).
user: NathanWycoff
issue_9659: state: open
title: Store initial random state on models?
body: I've sometimes wondered how to reproduce a model (or an error) if I did not provide a fixed random state.

Something we could consider is for every randomised estimator (or at least where `random_state` is not an int), to store something like `random_state_tuple_ = rng.get_state()`. This makes any saved model reproducible given the same data.

Given a saved estimator `est`, one can do:
```
est2 = clone(est)
rng = np.random.RandomState()
rng.set_state(est.random_state_tuple_)
est2.set_params(random_state=rng)
```
to get an identical estimator. Not particularly user friendly, but at least this makes it possible.

An alternative is storing `intial_random_state_ = copy.deepcopy(rng)`, being more user-friendly, as long as the user doesn't then accidentally generate something from it and so change it.

What do others thing?

Related issue (I think): #9658
user: jnothman
issue_9631: state: open
title: Feature addition: Pipeline comparison 
body: Hi There,

During our research, we keep having to compare different pipelines or feature sets, and comparing improvement in numerical accuracy is not the best way for few reasons. Testing for significance of improvement can be via non-parametric Friedman test and some postdoc analysis [[check our paper for real-world example of usage](http://www.biorxiv.org/content/early/2017/07/31/170381)].

I will be adding such a feature to one of my projects [`neuropredict`](https://github.com/raamana/neuropredict), which is aiming to make machine learning like eating ice cream for those who can't code :), and am thinking it might be better to include it in scikit-learn and use it from there. If the following sounds useful and good inclusion to sklearn, I will be happy send in a PR - just wanted to make sure there is no one else working on it already.

The results will look like something presented in [recent scikit-learn benchmarks paper](https://crossinvalidation.com/2017/08/22/quantitative-comparison-of-scikit-learns-predictive-models-on-a-large-number-of-machine-learning-datasets-a-good-start/).

![screen shot 2017-08-26 at 10 22 04 am](https://user-images.githubusercontent.com/3196357/29742348-8f6c2c8c-8a4b-11e7-845e-1dec0e366e03.png)


Simple usage I envision is
```python
from sklearn.pipeline import Pipeline
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.decomposition import PCA
from sklearn import Pipeline, compare_pipelines

X, y = get_dataset_placeholder()
pipe1 = Pipeline([('reduce_dim', PCA()), ('clf', SVC())])
pipe2 = Pipeline([('reduce_dim', PCA()), ('clf', RandomForestClassifier())])
signif_matrix, comparison_report = compare_pipelines([pipe1, pipe2], X, y)
```


Rough implementation would look like:
```python
def compare_pipelines(pipe_list, X, y, cv_choice=None, perf_metrics = None):
    """ Compares two pipelines and whether one is significantly better than other.

    Performs a non-parametric comparison of predictive performance of two pipelines
    evaluated from bootstrapped distributions of cross-validated performance.

    """

    num_pipelines = len(pipe_list)
    if num_pipelines < 2:
        raise ValueError('Atleast two pipelines are required for comparison.')

    if cv_choice is None:
        cv = ShuffleSplit(test_size=0.5, n_splits=250, random_state=1)

    if perf_metrics is None:
        perf_metrics = ['accuracy_score', 'balanced_accuracy', 'auc', 'f1_score']

    pipe_scores = np.empty([num_pipelines, 1])
    for counter, pipe in enumerate(pipe_list):
        # this would be replaced later by local implementation, wherein we control
        # which train/test sets pipelines will be trained and tested on for better estimation
        pipe_scores[counter] = cross_validate(pipe, X, y, 
                                    scoring = perf_metrics,
                                    cv = cv_choice)

    signif_matrix, comparison_report = non_parametric_friedman(pipe_scores, posthoc= 'nemenyi')
    
    return signif_matrix, comparison_report

```

cc @amueller @GaelVaroquaux 
user: raamana
issue_9630: state: open
title: Feature Request: Pipelining Outlier Removal
body: I wonder if we could make outlier removal available in pipelines.

I tried implementing it for example using the IsolationForest but so far I couldn't solve it and I know why.

The problem boils down to `fit_transform` only returning a transformed `X` this suffices in the vast majority of cases, since we typically only throw away columns (think of a PCA). However, using outlier removal in a pipeline, we need to throw away rows of `X` **and** `y` during training and do nothing during testing. This is not supported so far. Essentially, we would need to turn the `predict` function into some kind of `transform` function during training.

Investigating the pipeline implementation shows, that `fit_transform`is called if present during the fitting part of the pipeline, rather than `fit(X, y).transform(X)`. Particularly, in a cross validation `fit_transform` is only called during training. This would be perfect for outlier removal. However, it remains to do nothing in the test step. But to this end we can simply implement a "do-nothing" `transform`-function.

The most direct way to implement this, would be an API-change of the TransformerMixin-class,  unfortunately.

So my questions are:

Would it be interesting to contain feature removal in pipelines?
Are there other more suitable ideas of implementing this feature in a pipeline?

If the content of this question is somehow inapropriate (e.g. since I'm only an active user, not an active developer of the project) or at the wrong place, feel free to remove the thread.



user: datajanko
issue_9626: state: open
title: Trees with MAE criterion are slow to train
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
when I use 'mae' criterion for the model extratreesregressor, training for a long time,  it's seems lead to an endless training.   there have no problem for mse
I find not only me hava this problem. I hava tried two version (0.18 and  0.19.X) , but no used .

https://www.kaggle.com/c/allstate-claims-severity/discussion/24293


#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->
```python
from sklearn.ensemble import ExtraTreesRegressor
rfr = ExtraTreesRegressor(n_estimators=100,
max_features=0.8,  
criterion='mae', 
max_depth=6, 
min_samples_leaf=200,
 n_jobs=-1,
 random_state=17, 
verbose=0)
 mod = rfr.fit(train[distilled_features], train['y'])
```
#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
can normal training in my model when mae used

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
aways traing in fit step

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
Darwin-16.1.0-x86_64-i386-64bit
Python 3.6.1 |Anaconda 4.4.0 (x86_64)| (default, May 11 2017, 13:04:09)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
NumPy 1.12.1
SciPy 0.19.0
Scikit-Learn 0.19.X

<!-- Thanks for contributing! -->

user: jiangfeng
issue_9624: state: open
title: cohen's kappa can't handle single label cases
body: cohen_kappa_score assumes that the data is going to have multiple labels, although most time's that is the case for training a classifier that may not always be the case for testing, like for example if you are just evaluating a classifier over many datasets and one of them only has one class data. Sure this evaluation is kind of useless but this will happen when running a classifier through unknown datasets where you are trying to estimate classification performance.

Here how the problem could be reproduced:

```
from metrics import cohen_kappa_score
cohen_kappa_score([0,0],[0,0])
```

The above results in a nan, when in reality it should throw an error, or return a 1 with a warning. The one is because after all the classifier we are evaluating in this case predicted perfectly.
user: julian-ramos
issue_9615: state: open
title: [WIP] Introducing SoftImpute FactorizationImputer for scikit-learn
body: Continuing work from #4237 and #2387. 

Rather than minimizing the funcSVD-type lost functions specified in the previous attempts, this PR follows advice from @amueller and @GaelVaroquaux , and imports SoftImpute from [FancyImpute](https://github.com/hammerlab/fancyimpute) instead. 

Here is the new cost function being minimized:

![image](https://user-images.githubusercontent.com/1597013/29642894-342457c8-8820-11e7-9acc-93c1e9e27f40.png)


#### What this implements/fixes

A new transformer `sklearn.preprocessing.FactorizationImputer`, that uses tests/structure originally created by the previous author as part of #4237

#### What I'd like to hear back

1. Any general feedback on how to improve this.
2. How I can add the fancyimpute dependency to scikit-learn

#### Other comments

Thanks to @sergeyf and iskandr, the authors who created this initial implementation of SoftImpute in python.

Here is [the original paper](http://web.stanford.edu/~hastie/Papers/mazumder10a.pdf) where the SoftImpute algorithm was introduced. Here is [another helpful introduction](https://web.stanford.edu/~hastie/swData/softImpute/vignette.html) to how the algorithm works.

user: jeh0753
labels: Needs Decision: Requires decision
issue_9610: state: open
title: ParametersGrid parameters combinations representation
body: **Description**
Trying to define parameter grid for GridSearchCV I collided into a problem of handy representation of all possible combinations of parameters.
List of dictionaries or simple dictionary look rather cumbersome and doesn't provide easy way to aggregate some parameters.
Even taking this [example](http://scikit-learn.org/stable/modules/grid_search.html#exhaustive-grid-search) from documentation: 
```
param_grid = [
    {'C': [1, 10, 100, 1000], 'kernel': ['linear']},
    {'C': [1, 10, 100, 1000], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']}
]
```
Why not make the following structure:
param_grid always will be a dictionary. Each key of this dictionary that starts with double underscore means  aggregated parameter, and value besides this key — list of tuples:
- first entry of  tuple - the value of the parameter on which the aggregation was performed
- second entry of tuple - dictionary of parameters allowed with this certain aggregation value.
So same example will look as  follows:
```
 param_grid = {
        'C': [1,10,100,1000],
        '__kernel': 
        [
            ('linear', {}),
            ('rbf', { 'gamma': [0.001, 0.0001] } 
        ]
    }
```
In order to solve the problem of duplication possible values because of  partial intersection,  we also can  combine the collections under the same keys that located  both inside and outside of  some group. Another example(based on example from 'Introduction to ML with Python') might look as follows:
```
# before
param_grid = [
    {'classifier': [SVC()], 'preprocessing': [RobustScaler(),StandardScaler(), None],
    'classifier__gamma': [0.001, 0.01, 0.1, 1, 10, 100],
    'classifier__C': [0.001, 0.01, 0.1, 1, 10, 100]},
    {'classifier': [LogisticRegression()],
    'preprocessing': [RobustScaler(), None], 'classifier__penalty':['l1','l2']}
]

# after
param_grid = {
    'classifier__C': [0.001, 0.01, 0.1, 1, 10, 100],
    'preprocessing': [RobustScaler(), None],
    '__classifier': [
        (SVC(), { 'classifier__gamma': [0.001, 0.01, 0.1, 1, 10, 100]},
             'preprocessing':[StandardScaler()]),
        (LogisticRegression(),{ 'classifier__penalty':['l1','l2'] })
    ]
}
```
user: JointEntropy
issue_9608: state: open
title: deprecate biclustering?
body: Some analysis of our source code use suggests neither `SpectralBiclustering` nor `SpectralCoclustering` are known to be in public use. They have design issues that have not been addressed in many years: #2484. Do people use these, anecdotally?
user: jnothman
labels: Needs Decision: Requires decision
issue_9606: state: open
title: feature_importances_ should be a method in the ideal design
body: This issue is not meant to be very practical, just a place to share my thoughts.

I believe `feature_importances_` should have been designed as `get_feature_importances()` (which is, perhaps, funny because I think the `get_feature_names` design is pretty broken too), for the following reasons:
* calculating feature importances can be costly, and should not (and is not in some cases) be calculated at `fit` time unnecessary
* there are often multiple ways to calculate feature importances (as simply as choice of norm for `coef_`), and (as long as they depend on the same sufficient statistics) the user may fairly not decide which is appropriate until after `fit`. Thus `get_feature_importances` could have parameters to choose its method. Meta-estimators such as `SelectFromModel` and `RFE` currently have parameters for how they should interpret `coef_` as feature importances, but really these are parameters that should be passed to the linear model's `get_feature_importances`; the model itself should know how to summarise its `coef_`, and doing so gets more complicated once we have multi-output `coef_`.
* it is semantically different from other attributes, not being a sufficient statistic upon which basis the estimator makes predictions

I don't think there is currently sufficient motivation to change, but I could be persuaded.

Ping @kmike?
user: jnothman
issue_9603: state: open
title: least_angle.py in lars_path - shapes not aligned on properly formatted data with specific alpha
body: Encountered a similar error to https://github.com/scikit-learn/scikit-learn/issues/5873 - when running the following code:

`a_selection = RandomizedLasso(alpha=0.025, normalize=False, n_jobs=1, random_state=42)
 a_selection.fit(X=x_sub, y=y_sub)`
    
I can't share the data as it's from clinical trials, but what I have noticed is that the error disappears (for this particular fit) when I remove the alpha parameter. The code takes 2 days to complete so I am worried a different alpha will break a different dataset input. The data frame is properly formatted, the row number fits the labels and there are no NaNs. The error I get is:

`~/miniconda3/lib/python3.6/site-packages/sklearn/linear_model/randomized_l1.py in fit(self, X, y)
    110             n_jobs=self.n_jobs, verbose=self.verbose,
    111             pre_dispatch=self.pre_dispatch, random_state=self.random_state,
--> 112             sample_fraction=self.sample_fraction, **params)
    113 
    114         if scores_.ndim == 1:`

`~/miniconda3/lib/python3.6/site-packages/sklearn/externals/joblib/memory.py in __call__(self, *args, **kwargs)
    281 
    282     def __call__(self, *args, **kwargs):
--> 283         return self.func(*args, **kwargs)
    284 
    285     def call_and_shelve(self, *args, **kwargs):`

`~/miniconda3/lib/python3.6/site-packages/sklearn/linear_model/randomized_l1.py in _resample_model(estimator_func, X, y, scaling, n_resampling, n_jobs, verbose, pre_dispatch, random_state, sample_fraction, **params)
     52                 verbose=max(0, verbose - 1),
     53                 **params)
---> 54             for _ in range(n_resampling)):
     55         scores_ += active_set
     56 `

`~/miniconda3/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in __call__(self, iterable)
    756             # was dispatched. In particular this covers the edge
    757             # case of Parallel used with an exhausted iterator.
--> 758             while self.dispatch_one_batch(iterator):
    759                 self._iterating = True
    760             else:`

`~/miniconda3/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in dispatch_one_batch(self, iterator)
    606                 return False
    607             else:
--> 608                 self._dispatch(tasks)
    609                 return True
    610 `

`~/miniconda3/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in _dispatch(self, batch)
    569         dispatch_timestamp = time.time()
    570         cb = BatchCompletionCallBack(dispatch_timestamp, len(batch), self)
--> 571         job = self._backend.apply_async(batch, callback=cb)
    572         self._jobs.append(job)
    573 `

`~/miniconda3/lib/python3.6/site-packages/sklearn/externals/joblib/_parallel_backends.py in apply_async(self, func, callback)
    107     def apply_async(self, func, callback=None):
    108         """Schedule a func to be run"""
--> 109         result = ImmediateResult(func)
    110         if callback:
    111             callback(result)`

`~/miniconda3/lib/python3.6/site-packages/sklearn/externals/joblib/_parallel_backends.py in __init__(self, batch)
    324         # Don't delay the application, to avoid keeping the input
    325         # arguments in memory
--> 326         self.results = batch()
    327 
    328     def get(self):`

`~/miniconda3/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in __call__(self)
    129 
    130     def __call__(self):
--> 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
    132 
    133     def __len__(self):`

`~/miniconda3/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in <listcomp>(.0)
    129 
    130     def __call__(self):
--> 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
    132 
    133     def __len__(self):`

`~/miniconda3/lib/python3.6/site-packages/sklearn/linear_model/randomized_l1.py in _randomized_lasso(X, y, weights, mask, alpha, verbose, precompute, eps, max_iter)
    171                                       copy_Gram=False, alpha_min=np.min(alpha),
    172                                       method='lasso', verbose=verbose,
--> 173                                       max_iter=max_iter, eps=eps)
    174 
    175     if len(alpha) > 1:`

`~/miniconda3/lib/python3.6/site-packages/sklearn/linear_model/least_angle.py in lars_path(X, y, Xy, Gram, max_iter, alpha_min, method, copy_X, eps, copy_Gram, verbose, return_path, return_n_iter, positive)
    442 
    443                 # TODO: this could be updated
--> 444                 residual = y - np.dot(X[:, :n_active], coef[active])
    445                 temp = np.dot(X.T[n_active], residual)
    446 `

`ValueError: shapes (49,17) and (16,) not aligned: 17 (dim 1) != 16 (dim 0)`

Versions:
Linux-4.10.0-32-generic-x86_64-with-debian-stretch-sid
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:09:58)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
NumPy 1.12.1
SciPy 0.19.1
Scikit-Learn 0.18.2

Happy to provide any additional information if needed.

Lastly, I get a message that RandomizedLasso with be deprecated - what will replace it's functionality? Setting n_jobs parameter to anything beyond 1 breaks the code, which was already reported - hope the replacement will fix that.

Many thanks!
user: ghost
issue_9601: state: open
title: Tutorial on how to create a new kernel?
body: I am trying to implement my own kernel for Gaussian Processes regression.

In order to do so I tried to reproduce the structure of the common kernels in sklearn.gaussian_process.kernels but with no success at the moment. When I try to do regression and optimize the hyperparameters, something like:

    import sklearn
    import my_own_kernel

    k = my_own_kernel(Parameters)
    # GP optimization
    gp = sklearn.gaussian_process(kernel=k, n_restarts_optimizer=0)
    gp.fit(Xtr, Ytr)

I have always a bunch of errors. 
I have looked for a tutorial that explains how to build a new kernel that is compatible with the GP regression routines especially for the optimization of the hyperparameters but I did not find any. Something like [http://scikit-learn.org/stable/developers/contributing.html#rolling-your-own-estimator][1] but specific for kernels.

I am wondering if such tutorial exists and I missed it or if there is a reason why it is missing. In case it does not exist can it be made?

If this is not the right place to ask this kind of questions, please, point out where I should write. Thank you in advance for the help.


  [1]: http://scikit-learn.org/stable/developers/contributing.html#rolling-your-own-estimator
user: DiegoRomeres
issue_9585: state: open
title: Add pass classes argument to scorers and allow_subset_labels in metrics
body: Scorers pass classes from estimators into the labels argument of the metrics.
allow_subset_labels added to all the metrics allowing scoring on a subset of labels.

Issues: #6231 #8100 
user: thismlguy
labels: Needs Decision: Requires decision
issue_9581: state: open
title: [WIP] Gaussian Mixture Model with Missing Data Estimation and Imputation Support
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
<!-- Example: Fixes #1234 -->
Fixes: [9268](https://github.com/scikit-learn/scikit-learn/issues/9268)
However, please note that the implementation initially proposed in  [9268](https://github.com/scikit-learn/scikit-learn/issues/9268) has been replaced with the implementation as proposed in the primary reference mentioned below.

#### What does this implement/fix? Explain your changes.
This PR will implement a Gaussian Mixture Model with support for missing (NaN) data. The support will be available both for direct estimation of parameters in the presence of NaN or to impute missing values to be used with other estimators. The primary reference for this implementation will be [Ghahramani, Zoubin, and Michael I. Jordan. "Supervised learning from incomplete data via an EM approach." Advances in neural information processing systems. 1994](http://papers.nips.cc/paper/767-supervised-learning-from-incomplete-data-via-an-em-approach.pdf). The implementation will also draw heavily from the book [Schafer, Joseph L. Analysis of incomplete multivariate data. CRC press, 1997.](https://books.google.com/books?id=3TFWRjn1f-oC&dq=schafer+incomplete+multivariate+data&lr=&source=gbs_navlinks_s), especially in terms of implementing the EM algorithm in the presence of missing data. A further reference is Little, R. J. A., and D. B. Rubin. "Statistical Analysis with Missing Data." (2002).

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: ashimb9
issue_9553: state: open
title: Use median instead of mean when constructing RandomForestRegressor
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
I am using a RandomForestRegressor for a task where it seems I would benefit from having the leaf node values defined by the median over its samples instead of the mean over its samples, due to the presence of outliers. This functionality currently isn't present in scikit-learn, and I would like to add it (either as a PR, or just locally on my machine).

If I'm going to do this, I could a pointer as to how to do it. Specifically I am having trouble finding the line of code where sklearn currently takes the mean over values in a leaf. I'm aware that it's somewhere within the code for a decision tree. If I found that, I can probably handle the rest.

#### Steps/Code to Reproduce
```python
from sklearn.ensemble import RandomForestRegressor
help(RandomForestRegressor)
```
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

#### Desired API

I could just hack something in, if I'm doing it only for myself locally. If I'm going to write this as a change to the RFR API, then maybe something like:

```python
from sklearn.ensemble import RandomForestRegressor
import numpy as np
random_forest = RandomForestRegressor(reduction_op=np.median)  # default np.mean
```

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
Linux-3.10.0-514.10.2.el7.x86_64-x86_64-with-centos-7.3.1611-Core
('Python', '2.7.13 | packaged by conda-forge | (default, May  2 2017, 12:48:11) \n[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]')
('NumPy', '1.12.1')
('SciPy', '0.19.0')
('Scikit-Learn', '0.18.1')

<!-- Thanks for contributing! -->

user: mcgibbon
issue_9532: state: open
title: [WIP] Add classes parameter to all classifiers
body: A WIP PR, ref: #8100 and #9290 

Added classes to following classifiers:

- GaussianNB
- BernoulliNB
- MultinomialNB
- DecisionTreeClassifier
- ExtraTreeClassifier
- RandomForestClassifier
- ExtraTreesClassifier
- GradientBoostingClassifier [WIP]
- LinearSVC
- SVC (on hold)
- LogisticRegression
- LogisticRegressionCV

To be done:
- Others classifiers
- sparse input support (svm, gbm, etc)
user: thismlguy
labels: Needs Decision: Requires decision
issue_9460: state: open
title: Overfitting in PLSRegression
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->

It looks like PLSRegression is fitting noise really well

I've put together a null dataset simulating of the noise that I'd typically expect out of my dataset.
(See here for [notebook](https://github.com/mortonjt/benchmarkThis/blob/master/ipynb/part10_null_pls.ipynb))

Specifically, I generated the following dataset

![image](https://user-images.githubusercontent.com/4184797/28706554-ae1052be-7329-11e7-9ac3-c7acc3d75d53.png)

And I get the following PLS fit

![image](https://user-images.githubusercontent.com/4184797/28706558-b5334b78-7329-11e7-9ddc-52c4a72138f9.png)

Note, that I have tested this on simulated dataset with known signal, and it was able to recover the real signal.  However, it seems like PLSRegression is very prone to overfitting when tested on pure noise.

This is well-known in the literature - so its probably not a problem with the code.  But it would be really nice if there could be some good cross-validation statistics, or other methods that could protect against overfitting.  Is there any interest in including these sorts of methods / statistics?  This [paper](https://pdfs.semanticscholar.org/5eba/eae792915573295a5cbe516e342bce8b493e.pdf) lists a few of the statistics for measuring this sort of performance

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
Darwin-15.6.0-x86_64-i386-64bit
Python 3.5.3 |Continuum Analytics, Inc.| (default, Mar  6 2017, 12:15:08) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
NumPy 1.13.0
SciPy 0.19.0
Scikit-Learn 0.18.1

<!-- Thanks for contributing! -->

user: mortonjt
issue_9453: state: open
title: PLS* should not transform on y
body: Transforming on y doesn't conform with our API. Also it returns X, y which again doesn't conform with our API right?

Copying @bthirion's comment here -


@agramfort, I think that Y could be avoided in the transform for PLSRegression, not for PLS Canonical -- that is more like CCA. However, this probably implies a thorough refactoring of the code.

(Making an issue from a stalled PR #9160)
user: raghavrv
issue_9443: state: open
title: Add instance-level calls to estimator_checks for meta-estimators
body: We recently added the ability to run ``estimator_checks`` on instances.
We should add tests for the meta-estimators running ``estimator_checks`` on common settings, including
* [ ] pipeline
* [ ] feature union
* [ ] GridSearchCV(Estimator())
* [ ] GridSearchCV(Pipeline(...))
* [ ] RandomizedSearchCV(...)

And then possibly additional tests for anything that is currently not tested by the common tests. I think they raise SkipTest warnings, but I'm not sure? There's at least ``SparseEncode``, RFECV, SelectFromModel, and probably more, that are not tested. Maybe CalibratedClassifierCV, VotingClassifier?

If someone wants to work on this, they can start with the five above and check out which other estimators are not tested and post a list here...
user: amueller
labels: module:test-suite: everything related to our tests
issue_9439: state: open
title: [WIP] Make knn kernel undirected.
body: This is a very simple way of making the `knn` kernel matrix symmetric. However, this slightly changes the meaning of `n_neighbors`: the _actual_ number of neighbors may vary from `n_neighbors` to `2 * n_neighbors` in the worst case.

 - [ ] Update tests after #5893 is merged.

Fixes #8008.
user: musically-ut
labels: Needs Decision: Requires decision
issue_9429: state: open
title: Benchmark NMF and SGDClassifier on MKL vs OpenBlas
body: At scipy the intel python distribution people cited a speedup of ~10x for NMF and SGDClassifier on MKL vs OpenBlas. It would be great if someone could try to reproduce, and find the bottlenecks.
user: amueller
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_9414: state: open
title: Multioutput-multiclass estimators have broken score method.
body: It looks to me like the decision trees use ``accuracy_score`` for their ``score`` but ``accuracy_score`` doesn't document that it's supporting ``multiclass-multioutput`` which the trees do.

~~I guess the ``y_true == y_pred`` works in this case, but it should be documented.
There's no list of scores supporting multiclass-multioutput in the docs, and that should be fixed, too.~~

Update:
Calling ``score`` in this case errors :-/
user: amueller
issue_9413: state: open
title: [MRG] add stratify and shuffle variants for GroupKFold
body: Supersedes  #5396

Adds an option "method" to GroupKFold to change the way groups are distributed over folds. Current default is to balance the sizes of the folds. This adds the alternative of stratifying on the y variable, or shuffling the groups to randomize the folds they end up in.
user: andreasvc
labels: Needs Decision: Requires decision
issue_9400: state: open
title: PCA, LDA, unexpected explained_variance_ratio
body: ```python
from sklearn import datasets
from sklearn.decomposition import PCA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

iris = datasets.load_iris()

X = iris.data
y = iris.target
target_names = iris.target_names

#### dimensionality reduction using PCA
pca = PCA(n_components=2)
X_r = pca.fit(X).transform(X)

#### Percentage of variance explained for each components
print('PCA: explained variance ratio (first two components): %s'
      % str(pca.explained_variance_ratio_))

#### dimensionality reduction using LDA
lda = LinearDiscriminantAnalysis(n_components=2)
X_r2 = lda.fit(X, y).transform(X)

print('LDA: explained variance ratio (first two components): %s'
      % str(lda.explained_variance_ratio_))
```
#### Expected Results
The first componet of the PCA has a larger variance ratio than that from the first componet from LDA.

#### Actual Results
PCA: explained variance ratio (first two components): [ 0.925  0.053]
LDA: explained variance ratio (first two components): [ 0.991  0.009]

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->

Darwin-14.5.0-x86_64-i386-64bit
Python 3.5.3 |Anaconda custom (x86_64)| (default, Mar  6 2017, 12:15:08) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
NumPy 1.12.1
SciPy 0.19.1
Scikit-Learn 0.18.2

<!-- Thanks for contributing! -->
user: yaohualiu
issue_9390: state: open
title: extract_patches_2d max_patches param behaviours
body: Hi, initially this started as SO question. However, now I really think is might be considered as an issue. 

I'm using sklearn version 0.18.2

In particular, I'm using this function for extracting image regions: http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.image.extract_patches_2d.html

This function has one funny argument, namely max_patches which says how many patches function should extract from the image.

According to the source code, lines 228-238, if the number of requested samples is higher than the number of patches available in the image, an exception is raised:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/image.py

In my image set I have images of different sizes: from really tiny images to full HD ones. And now this is a problem as when I try to request, say, 1e6 patches, it throws an error on small images.

The workaround, I found is as follows. Is it alright is just paste link with my SO answer here?
https://stackoverflow.com/questions/45146347/extract-patches-2d-max-patches-param-behaviours/

If no, I can correct it. Long, story short to get what I want I had to use a private module function, namely `_compute_n_patches`

Should this be considered as an issue? When you request the amount of patches that is greater than available, you would expect to get this amount of patches at maximum. If this amount of patches is unavailable,  why throw an exception? 

What am I missing? 

Thanks
user: sofeikov
labels: Needs Decision: Requires decision
issue_9388: state: open
title: Test docstrings for parameters are equal
body: We recently merged a test that checks consistency between parameters in function signatures and their docstrings using `sklearn.utils.testing.check_docstring_parameters`. I would like to have a function in `sklearn.utils.testing` which similarly makes use of [`numpydoc`](https://github.com/numpy/numpydoc/) to check that parts of docstrings are identical among a set of objects. I would expect this helper to eventually be contributed back to numpydoc.

It might look something like:
```python
def assert_consistent_docs(objects,
                           include_params=None, exclude_params=None,
                           include_attribs=None, exclude_attribs=None
                           include_returns=None, exclude_returns=None):
    """

    Checks if types and descriptions of parameters, etc, are identical across
    objects. ``object``s may either be ``NumpyDocString`` instances or objects
    (classes, functions, descriptors) with docstrings that can be parsed as
    numpydoc.

    By default it asserts that any Parameters/Returns/Attributes entries having the
    same name among ``objects`` docstrings also have the same type
    specification and description (ignoring whitespace).

    ``include_*`` and ``exclude_*`` parameters here are mutually exclusive,
    and specify a whitelist or blacklist, respectively, of parameter, attribute or
    return value names. May be '*' to include/exclude all from that section.
    """
    ... do stuff ...
```

Then we could call it with:
```
assert_consistent_docs([sklearn.metrics.precision_recall_fscore_support,
                        sklearn.metrics.precision_score,
                        sklearn.metrics.recall_score,
                        sklearn.metrics.f1_score,
                        sklearn.metrics.fbeta_score],
                       exclude_returns='*')
```
This will ensure (by making a test fail when the condition is not met) that all these related scoring functions have identical parameter descriptions (whitespace excepted) wherever they have identical parameters. (I've not actually checked whether this is or should be true of all these metric functions.) Most importantly, having such an assertion means we can rest assured that when we change the documentation of some parameter or return value, we will be forced to do so consistently.
user: jnothman
issue_9382: state: open
title: NN-Descent/KGraph for approximate knn
body: Efficient approximate kNN-graph construction, and efficient approximate kNN search using such a graph would be of significant potential benefit for scikit-learn. Such methods are independent of data dimension, and support generic similarity measures. The relevant papers, including benchmarks are:

Dong, Wei, Charikar Moses, and Kai Li. "Efficient k-nearest neighbor graph construction for generic similarity measures." Proceedings of the 20th international conference on World wide web. ACM, 2011.

Kiana Hajebi, Yasin Abbasi-Yadkori, Hossein Shahbazi, and Hong Zhang. 2011. Fast approximate nearest-neighbor search with k-nearest neighbor graph. In Proceedings of the Twenty-Second international joint conference on Artificial Intelligence - Volume Volume Two (IJCAI'11), Toby Walsh (Ed.), Vol. Volume Two.

Fu, Cong, Changxu Wang, and Deng Cai. "Fast Approximate Nearest Neighbor Search With Navigating Spreading-out Graphs." arXiv preprint arXiv:1707.00143 (2017).
 
Such methods are implemented by the kGraph library, and perform extremely well in the ANN-benchmarks:

https://github.com/erikbern/ann-benchmarks

The algorithms themselves are fairly straightforward. I have successfully implemented the kNN-graph construction algorithm in around 100 lines of cython (supported code for heaps no included). Performance, even for simple implementations such as mine is still very good. Is there interest in including an implementation of such techniques? If so I would be happy to provide code.

Ping @ogrisel @jnothman 

 


user: lmcinnes
issue_9377: state: open
title: New Feature :  Hierachy Decision Tree
body: We have classification problem with input state of dimension 1,000,000 (very large).
Current Decision Tree using greedy algorithm, it's impossible to get the optimal tree.

But, due to the nature of the data, the input features can "structured" in hiearchy 
and not all the 1,000,000 input feature needs to be evaluated at each level node.

Desire feature :  
As input of Decision Tree, specify the sub-list of columns(ie feature), we want to use
```
level 0 node:    [0, 1,0,1,0, 1]   (only some columns are evaluated in greedy algorithms)
level 1 node:    [0, 1,0,1,0, 1]   (only some columns are evaluated in greedy algorithms)
level 2 node:    [1, 1,1,1,1, 1]   (all are considered for the greedy algorithm).
```

For very large input dimension, it helps converging to the optimal tree, by using "extra knowledge".






user: arita37
issue_9360: state: open
title: use of Parallel class by OneVsRestClassifier versus BaggingClassifier
body: #### Description
I'm just getting started looking at performance for SVM, and I found the following code that bags SVM to greatly improve speed. I tried adding in the n_jobs=-1 param for OneVsRestClassifier and for BaggingClassifier, but on my PC as well as a VM I'm using, multiple CPUs are only used in BaggingClassifier while OneVsRestClassifier only uses one. I have a feeling this represents my lack of understanding with the SVC algorithm, and I'd be appreciative of any help you could offer! In the mean time I'll try to get a handle on the SVC algorithm as well as the Parallel class that n_jobs uses
#### Steps/Code to Reproduce
Based on https://stackoverflow.com/questions/31681373/making-svm-run-faster-in-python
```import time
import numpy as np
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier
from sklearn import datasets
from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import SVC

iris = datasets.load_iris()
X, y = iris.data, iris.target

# Single SVC
X = np.repeat(X, 100, axis=0)
y = np.repeat(y, 100, axis=0)
start = time.time()
clf = OneVsRestClassifier(SVC(kernel='linear', probability=True, class_weight='balanced'), n_jobs=-1)
clf.fit(X, y)
end = time.time()
print "Single SVC", end - start, clf.score(X,y)

# Bagging SVC
n_estimators = 10
start = time.time()
clf = OneVsRestClassifier(BaggingClassifier(SVC(kernel='linear', probability=True, class_weight='balanced'), max_samples=1.0 / n_estimators, n_estimators=n_estimators, n_jobs=-1))
clf.fit(X, y)
end = time.time()
print "Bagging SVC", end - start, clf.score(X,y)
```

#### Expected Results
I expect "Single SVC" and "Bagging  SVC" both to use all my CPUs. 

#### Actual Results
The "Single SVC" code runs on one CPU, and the "Bagging SVC" code runs on all my CPUs. Relatedly, when I try to add n_jobs=-1 to the OneVsRestClassifier of Bagging SVC, I get the warning:

sklearn/ensemble/bagging.py:337:
Multiprocessing-backed parallel loops cannot be nested, setting n_jobs=1

#### Versions
>>> import platform; print(platform.platform())
Linux-4.4.0-81-generic-x86_64-with-Ubuntu-16.04-xenial
>>> import sys; print("Python", sys.version)
('Python', '2.7.12 (default, Nov 19 2016, 06:48:10) \n[GCC 5.4.0 20160609]')
>>> import numpy; print("NumPy", numpy.__version__)
('NumPy', '1.13.0')
>>> import scipy; print("SciPy", scipy.__version__)
('SciPy', '0.19.0')
>>> import sklearn; print("Scikit-Learn", sklearn.__version__)
('Scikit-Learn', '0.17.1')

user: waltermblair
issue_9341: state: open
title: KBinsDiscretizer: allow nans
body: Missing values, represented as NaN, could be treated as a separate category in discretization. This seems much more sensible to me than imputing the missing data then discretizing.

In accordance with recent changes to other preprocessing, NaNs would simply be ignored in calculating `fit` statistics, and would be passed on to the encoder in `transform`. I can't recall if we're handling this sensibly in OneHotEncoder yet...
user: jnothman
issue_9337: state: open
title: KBinsDiscretizer: Automatic determination of number of bins
body: One small extension to KBinsDiscretizer is to allow the number of bins to be guessed by the estimator, using one of the strategies supported by [`np.histogram`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.histogram.html). We very possibly don't want to implement all of the options, but `fd`, `sturges` and `auto` might be appropriate.

However, I'm not actually sure how useful these estimates are in discretization, when they have been designed for visualisation. So a contribution would be best accompanied by an example that showed that this automatic determination was better for machine learning than a fixed number of bins across all features.
user: jnothman
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_9334: state: open
title: [MRG] partial_fit for BayesianGaussianMixture
body: #### Reference Issue
Implements #8714 Online Inference for BayesianGaussianMixture

#### What does this implement/fix? Explain your changes.
Implements a partial_fit() method for BayesianGaussianMixture models, which allows for online or minibatch inference.  It simply sets the posterior from the previous fit as the new prior,  a method known as [Streaming Variational Inference](http://proceedings.mlr.press/v45/Huynh15.pdf).  Under very mild conditions, the expectation of the posterior on each batch forms an unbiased estimate of the posterior on the whole data, so after going through all batches the models will be very similar.  Due to the stochastic nature of the fitting process, it often reaches a lower Evidence Lower Bound than fitting the model to the whole dataset, because it is less likely to get stuck in local minima during optimization.

I included an example identical to the one comparing Bayesian and non-Bayesian mixture models, but with many more samples to take advantage of the speed increase.

#### Any other comments?
Was not sure what tests were required/useful, since it just sets a few parameters and then makes use of the existing fit() function which has its own tests.  I also wanted add the image from the example to the mixture docs but couldn't figure out how auto_examples worked.

- [x] main implementation
- [x] example
- [x] documentation
- [x] tests
user: jengelman
issue_9305: state: open
title: Access "sample_weight" in adaboost
body: Hello,

For a research paper I need to access the weights calculated by the Adaboostclassifier. Would it be possible to add the variable "sample_weight" as an attribute so I could publish code working with scikit-learn out of the box ?

Thank you,

Pierre
user: pierthodo
issue_9297: state: open
title: [feature request] GroupShuffleSplit version to balance group sizes 
body: #### Description
`GroupShuffleSplit` splits vectors into groups based on the group fractions unlike similar generators that balance the number of samples.

Suggestion: provide a generator that makes some effort to balance the number of samples.

#### Possible solution prototype
```python
from collections import Counter
def get_group_split(groupvector, test_size = 0.2):
    if type(test_size) is float:
        test_size = round(test_size*len(groupvector))
    gr_counts = Counter(groupvector)
    gr_keys = np.random.choice(np.asarray(list(gr_counts.keys())), size = len(gr_counts), replace=False)

    ind = np.argmin(abs(np.cumsum(np.array([gr_counts[x] for x in gr_keys])) - test_size))
    if ind > (len(gr_counts) -1):
        print("split index", ind)
        print(
            abs(np.cumsum(np.array([gr_counts[x] for x in gr_keys])) - test_size)
            )
        raise ValueError("Unable to split")
    train_keys = [kk for nn, kk in enumerate(gr_keys) if nn>ind]
    return train_keys, test_keys

# Example
## generate a skewed distribution of a grouping variable
p = 1.0/(1+np.arange(10))
p = p/sum(p)
grouping_var = np.random.choice(np.arange(10), 100, p=p)

def showcase_get_group_split(grouping_var, test_size = 0.2, n = 200):
    ts_sizes = []
    for _ in range(n):
        tr_k, ts_k = get_group_split(grouping_var, test_size = test_size)
        ts_sizes.append(sum([x in set(ts_k) for x in grouping_var]))
    #     break
    print("std of test set size:", np.std(ts_sizes))
    plt.hist(ts_sizes, bins=20)
    plt.xlim([0, 50])
    pass

showcase_get_group_split(grouping_var, test_size=0.2)


# compare to GroupShuffleSplit
ts_sizes = []
for _ in range(200):
    _, test_images = next(GroupShuffleSplit(test_size=0.2)._iter_indices(grouping_var, grouping_var, grouping_var))
    ts_sizes.append(test_images.shape[0])

print("std of test set size:", np.std(ts_sizes))
plt.hist(ts_sizes, bins=20)
plt.xlim([0, 50])
```

Standard deviation of the test size with the prototype reduces from about 11 to 6 in this example.

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->

Linux-4.2.0-42-generic-x86_64-with-debian-jessie-sid
Python 3.6.1 (default, Jun 11 2017, 12:35:03) 
[GCC 4.8.4]
NumPy 1.13.0
SciPy 0.19.0
Scikit-Learn 0.18.1

<!-- Thanks for contributing! -->

user: DSLituiev
issue_9290: state: open
title: [WIP] Adding classes argument to Gaussian Naive Bayes
body: Fixing issue #8100 

This PR involves adding a classes argument to the initialization of the gaussian naive bayes estimator. This would make the classes argument of partial_fit redundant as we'll no longer need to specify it every time.
user: thismlguy
labels: Needs Decision: Requires decision
issue_9285: state: open
title: Inverse Transform for Label Encoder with mixed strings and numbers returns only strings
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
With a LabelEncoder fitted with both string and numeric values, the inverse transform of that LabelEncoder will include only strings.

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->
```python
from sklearn.preprocessing import LabelEncoder
le=LabelEncoder().fit([1, 2, 'a', 'b'])
le.inverse_transform([0, 1, 2, 3])
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
```
array([1, 2, 'a', 'b'], dtype=object)
```
I understand that numpy is not ideal for dealing with non-numeric data, so I don't know what dtype the output SHOULD be, but I know that if the dtype is simply "object", then it will differentiate between the strings and numbers.
#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
```
array('1', '2', 'a', 'b'], dtype='<U11')
```
The array is no longer mixed type.
#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
Windows-8.1-6.3.9600-SP0
Python 3.5.3 |Anaconda 4.4.0 (64-bit)| (default, May 15 2017, 10:43:23) [MSC v.1900 64 bit (AMD64)]
NumPy 1.12.1
SciPy 0.19.0
Scikit-Learn 0.18.1


<!-- Thanks for contributing! -->

user: milags
labels: Needs Decision: Requires decision
issue_9268: state: open
title: Missing Data Imputation with a Gaussian Mixture Model using EM
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
I was wondering if there was interest in adding a new imputation strategy (or a new Imputer class) based on a Gaussian Mixture Model (GMM) using the EM or CEM algorithm. The implementation could be along the lines of:

1. [Ghahramani, Zoubin, and Michael I. Jordan. "Supervised learning from incomplete data via an EM approach." Advances in neural information processing systems. 1994.](http://papers.nips.cc/paper/767-supervised-learning-from-incomplete-data-via-an-em-approach.pdf)

2. [Ming Ouyang, William J. Welsh, Panos Georgopoulos; Gaussian mixture clustering and imputation of microarray data. Bioinformatics 2004; 20 (6): 917-923. doi: 10.1093/bioinformatics/bth007](https://academic.oup.com/bioinformatics/article/20/6/917/233425/Gaussian-mixture-clustering-and-imputation-of)

 3. [ Machine Learning: A Probabilistic Approach, Kevin Murphy (2012)](https://www.cs.ubc.ca/~murphyk/MLbook/) pp. 372-375. (although using a mixture rather than a single MVN in that example).

Further, Murphy's implementation of a Gaussian Mixture Model based imputation algorithm is also available [here](https://github.com/probml/pmtk3/blob/master/toolbox/LatentVariableModels/mixGauss/mixGaussImpute.m) (which I have not tested, by the way).

#### EDIT: I am working on a completely revised approach, which is now based on reference 1 above. Will post a working version soon.
user: ashimb9
issue_9267: state: open
title: Classification scorers requiring categorical predictions should not be applied to regressors
body: An error should be raised if a user runs cross validation with a scorer that is inapplicable to the base estimator. For instance, if the base estimator is a regressor, scoring='accuracy' should be disallowed when get_scorer is called, not when the score should be evaluated (which is the issue in #9264). However, we may want to allow roc auc to be calculated on a regressor, so there is don't nuance here
user: jnothman
labels: Hard: Hard level of difficulty
issue_9245: state: open
title: Extend ClassifierChain to multi-output problems
body: ClassifierChain currently supports multilabel classification. It should be straightforward to extend it to multi-output (as long as it only chains on `predict`) except for implementing `ClassifierChain.{predict_proba,decision_function}` which will take some care.
user: jnothman
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_9244: state: open
title: [WIP] add RBF kernel to MeanShift package and correct the website description 
body: 

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
<!-- Example: Fixes #1234 -->


#### What does this implement/fix? Explain your changes.

On the current MeanShift website (http://scikit-learn.org/stable/modules/generated/sklearn.cluster.MeanShift.html), in the "Parameters" section, it states that ”Bandwidth used in the RBF kernel”.  However, the RBF kernel is not implemented in the MeanShift package.  The current MeanShift package uses, and only uses the linear kernel.

I introduced one more parameter of this package that the user can choose to use linear or RBF kernel, and also implement the RBF kernal's way to calculate the weighted mean.

I also removed the description ”Bandwidth used in the RBF kernel” from the website.


#### Any other comments?
I tested the code using ./build_tools/travis/flake8_diff.sh, pyflakes, and pep8.  Did not see error messages.  But the tool "make" says "make: *** [test-code] Killed: 9".  I don't know if it means 9 tests have failed.  But my change should not affect the results of the calculations, because if the user does not specify the value of the new parameter "kernel", it uses the default value, linear kernel, which is how the current code implemented.   

Please advise how I can past all the tests using "make"

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: qiaonahu
issue_9179: state: open
title: [MRG+1] Implements SelectDimensionKernel
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue

Finishes up and closes https://github.com/scikit-learn/scikit-learn/pull/7070

Fixes #6868

#### What does this implement/fix? Explain your changes.

I just made some minor cosmetic changes to an example.

#### Any other comments?

The original PR already had two +1's by me and by @jmetzen . This should be good to merge once tests pass.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: MechCoder
issue_9173: state: open
title: [MRG] Add plotting module with heatmaps for confusion matrix and grid search results
body: Continues PR [8082](https://github.com/scikit-learn/scikit-learn/pull/8082)

Changes made:

- Added plot functions - plot_confusion_matrix, plot_gridsearch_results
- Updated examples - plot_rbf_parameters.py, plot_confusion_matrix.py
- added unit tests for new plot modules
user: thismlguy
issue_9169: state: open
title: [MRG] Dummy 2d fix
body: Another fix from the estimator tags branch. If ``y.shape == (n_samples, 1)`` master crashes.
user: amueller
issue_9141: state: open
title: Unnecessary memcopy in Tree.__setstate__
body: The nodes are stored as a contiguous C array of `Node` structs in the `sklearn.tree._tree.Tree` cython class. This is great as it allows to make (de)serialization fast.

However the `__setstate__` method issues a `memcopy` call from the data pointer of the passed numpy array for the node content.

I think this is not necessary, in particular, we could instead just store a pointer and keep a python reference to the wrapping array to prevent garbage collection by Python.

In particular getting rid of the `memcopy` would make it possible to use several Python processes (e.g. gunicorn, celey or PySpark workers) to share memory when computing predictions from the same random forest model with `joblib.load('/path/to/large_random_forest.pkl', mmap_mode='r')` on the same machine.

@arjoly @glouppe @larsmans @jnothman WDYT?
user: ogrisel
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_9136: state: open
title: Add keyboard interrupt handling into cython files (particular those that run long)
body: At PyParis @ogrisel and me and some others thought that using [cysignals](http://cysignals.readthedocs.io/en/latest/interrupt.html#basic-example) to catch keyboard interrupts would be cool. Unfortunately it's LGPL. But it's not that hard to do that ourselves:

https://stackoverflow.com/questions/16769870/cython-python-and-keyboardinterrupt-ignored

We "only" need to periodically check for it.
The main use-case for this for me is that I ran some line in Jupyter and then realized i did something wrong but it takes 5h to complete. Or it runs longer than I anticipated and I want to stop it earlier.
Right now I think I need to kill the kernel, but that might even still have the process running in the background eating ram and CPU, I'm not sure.

For me this was mostly an issue with liblinear and libsvm so far. Not sure where else.
user: amueller
issue_9134: state: open
title: LatentDirichletAllocation perplexity reporting inconsistency using (un)normalized document-topic matrix
body: 
#### Description
Printing perplexity values during training of LDA model results in perplexity values calculated using an unnormalized document-topic-matrix [returned from lda._e_step()]. However, calling lda.perplexity() on the same training data uses lda.transform(), which returns the normalized document-topic matrix, making the perplexity values reported inconsistent. 

#### Steps/Code to Reproduce
```python
from __future__ import print_function

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.datasets import fetch_20newsgroups

n_samples = 2000
n_features = 1000
n_topics = 10

dataset = fetch_20newsgroups(shuffle=True, random_state=1,
                             remove=('headers', 'footers', 'quotes'))
data_samples = dataset.data[:n_samples]

tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2,
                                max_features=n_features,
                                stop_words='english')
tf = tf_vectorizer.fit_transform(data_samples)

lda = LatentDirichletAllocation(n_topics=n_topics, max_iter=5,
                                verbose=1, evaluate_every=1,
                                learning_method='online',
                                learning_offset=50.,
                                random_state=0)
# observe printed perplexity values
print('Perplexity values reported during training:')
lda.fit(tf)

print('Perplexity using normalized doc-topic matrix returned by lda.transform():')
print(lda.perplexity(tf))  # calls lda.transform(tf), which normalizes doc-topic-matrix

print('Perplexity using unnormalized doc-topic matrix returned by lda._e_step(), as used in lda.fit():')
print(lda.perplexity(tf, lda._e_step(tf,False,False)[0]))
```


#### Actual Results
Perplexity values reported during training:
iteration: 1, perplexity: 734.8955
iteration: 2, perplexity: 676.4426
iteration: 3, perplexity: 650.0283
iteration: 4, perplexity: 634.7467
iteration: 5, perplexity: 624.8908

Perplexity using normalized doc-topic matrix returned by lda.transform():
840047.754815
Perplexity using unnormalized doc-topic matrix returned by lda._e_step(), as used in lda.fit():
624.890802119

#### Versions
Darwin-13.4.0-x86_64-i386-64bit
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 23 2015, 02:52:03) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
NumPy 1.12.0
SciPy 0.18.1
Scikit-Learn 0.18.1



user: nasrallah
issue_9113: state: open
title: Add class_weight feature to MLPClassifier
body: As far as I can tell the MLP classes don't have support for different class weights which are useful for imbalanced classes. Adding this should be easy as it just comes in in the batch gradient summation and normalization. If you think this is useful I can create a PR.
user: lucasgreene
issue_9107: state: open
title: LinearDiscriminantAnalysis, explained variance ratio superior to one
body: Hi !

I am performing PCA and LDA on several micro-array data, with a y label vector for various mutant and an X matrix for the genes expression (see below), with a code similar to the example provide on 
http://scikit-learn.org/stable/auto_examples/decomposition/plot_pca_vs_lda.html#sphx-glr-auto-examples-decomposition-plot-pca-vs-lda-py

All the PCA works well, but few LDA output an Explained variance ratio superior to one. I suspect some float error in the array return by the lda object created by the following code and poor data not allowing to get a good explained variance : 

        nb_class=nb_class_count(y)
        
        lda = LinearDiscriminantAnalysis(solver="eigen", shrinkage="auto", n_components=2)
        X_r2 = lda.fit(X, y).transform(X)       
        lda2 = LinearDiscriminantAnalysis(solver="eigen", shrinkage="auto")
        X_r3 = lda2.fit(X, y).transform(X) 


        print "nb_class", nb_class        
        print "lda.explained_variance_ratio_", lda.explained_variance_ratio_      
        print "lda2.explained_variance_ratio_", lda2.explained_variance_ratio_

Return : 
>nb_class 4
>lda.explained_variance_ratio_ [ 1.29840783  1.0474278 ]
>lda2.explained_variance_ratio_ [ 1.29840783  1.0474278   0.98598958]

Where X and y are : 

X [[-0.0381006  0.0107296 -0.0463756 ..., -0.520574   0.201947   0.372232 ]
 [-0.0800158  0.0292747  0.203857  ..., -0.405826   0.532436  -0.310816 ]
 [ 0.118346  -0.10846    0.0856565 ...,  0.414698  -0.464304  -0.308325 ]
 ..., 
 [-0.0624702 -0.145009   0.240819  ..., -0.366957  -0.518862  -0.379997 ]
 [-0.108163   0.114644   0.141642  ...,  0.0443211  2.91074    3.74194  ]
 [-0.169948  -0.158028   0.186845  ...,  0.289508   3.44199    2.14646  ]]

y [ 3.  3.  2. ...,  0.  0.  0.]


Linux-4.9.0-3-amd64-x86_64-with-debian-9.0
('Python', '2.7.13 (default, Jan 19 2017, 14:48:08) \n[GCC 6.3.0 20170118]')
('NumPy', '1.12.1')
('SciPy', '0.18.1')
('Scikit-Learn', '0.18.1')

Nonetheless, I become suspicious for all my results now. Is there a way to assess the exact number of the explained ratio ?

Tell me if you need any more code.




user: Cdk29
issue_9091: state: open
title: [WIP] Eleven point average precision
body: Continuation of #9017 (itself a continuation of #7356).

Adds an eleven-point interpolation method for average precision, as described in the IR book.

user: GaelVaroquaux
issue_9088: state: open
title: deprecate using fit without y in GridSearchCV
body: Currently y is only passed to fit if it is not None, I think we should always pass it, as that's the standard API.
I'm not sure if that could break user code, though. Not if they use ``check_estimator``... hm..
user: amueller
labels: Needs Decision - Close: Requires decision for closing
issue_9059: state: open
title: [MRG+1] Added support for sparse multilabel y for Nearest neighbor classifiers
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
<!-- Example: Fixes #1234 -->

Fixes #8057


#### What does this implement/fix? Explain your changes.

Added support for sparse multilabel y for nearest neighbor classifiers. Firstly, it checks if the input to fit is sparse + multilabel and converts it to a dense one for storing. Also, it stores another parameter indicating whether the original input was sparse + multilabel or not. Now, in predict, if this stored value is true, then it converts y_pred to sparse CSC.
Also, added tests for the same.

#### Any other comments?

This PR wraps up @dalmia 's work in #8096 
user: massich
issue_9014: state: open
title: [MRG] partial_fit implementation in TfidfTransformer
body: #### Reference Issue
Closes #7549

#### What does this implement/fix? Explain your changes.
This PR implements a partial_fit method for TfidfTransformer.
As discussed in the thread https://github.com/scikit-learn/scikit-learn/issues/7549#issuecomment-250955363 , the number of features should not change after the partial_fit call, so I only update the DF. In order to do that, I slightly changed the logic of the TfidfTransformer: it maintains the vector of document frequencies (df) and the number of documents n_sample, not the actual idf vector. Instead, the idf vector is calculated when needed.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: n0mad
issue_9008: state: open
title: Ability to cache FeatureUnion transformers
body: It seems reasonable to support a `memory` parameter to `FeatureUnion` like was recently added to `Pipeline` (#7990). It is valuable in the sense that parameters in some constituent transformers can be searched over while others are unchanged; those that are unchanged should not need to be re-fit from scratch.
user: jnothman
issue_9007: state: open
title: Suggestion: cache all Pipeline steps by default
body: PR #7990 implements a caching pipeline which caches all steps but the last. I haven't found much discussion on this topic specifically, so I can only speculate on why the last step is not cached. With this issue, I would like to make the case for caching all pipeline steps.

Arguments pro:
1. I would guess the last step is not cached because it is usually not a transformer. However, the last step may well be a transformer, for example a pipeline of preprocessing steps inside of a parent pipeline that ends in a non-transforming estimator.
2. Even if the the last step is not a transformer, there could be cases where the user would want to cache the last step.
3. If all steps are cached, it is easy to recreate the behaviour currently implemented in #7990 by simply putting all steps you want cached in a caching pipeline, and insert that pipeline into a non-caching pipeline for steps you don't want cached. Conversely, it is difficult to cache all steps in a pipeline with the current behaviour.
user: lsorber
issue_8994: state: open
title: DictionaryLearning should support more general constraints on the components 
body: #### Description
In the  <a href="https://hal.inria.fr/inria-00408716v1/document">reference paper</a>, arbitrary convex compact contraints can be imposed on the dictionary atoms, as long as one can compute the euclidean projection (e.g l2 ball, elastic-net ball, fused-lasso ball, etc.). For example, sparsity can be a crucial constraint if we know from prior knowledge that the dictionary atoms / components should be sparse.

#### Proposal
The idea would be to allow the `DictionaryLearning` and `MiniBatchDictionaryLearning` estimators take strings (e.g "enet ball") or function handles (e.g `compute_fused_lasso_projection(locals())`), to allow it customize the way the dictionary atoms are updated.

#### Waiting for
PR #9036 merged.

user: dohmatob
issue_8993: state: open
title: BaggingRegressor and BaggingClassifier should support multi-output targets
body: #### Description
It would be nice to have `BaggingRegressor` / `BaggingClassifier` support multi output targets.

### This code faults
```
import numpy as np
from sklearn.ensemble import BaggingRegressor

n, p, q = 100, 200, 40
X = np.random.randn(n, p)
y = np.random.randn(n, q)
y = np.random.randn(n, q)
BaggingRegressor().fit(X, y)
```
with error

```
ValueError: bad input shape (100, 40)
> /home/elvis/anaconda2/envs/py3k/lib/python3.6/site-packages/sklearn/utils/validation.py(576)column_or_1d()
    574         return np.ravel(y)
    575 
--> 576     raise ValueError("bad input shape {0}".format(shape))
```

#### Proposed fix
```
BaggingRegressor(multi_output=True).fit(X, y)
```
user: dohmatob
issue_8987: state: open
title: Bug - wrong activations and delta length in BaseMultilayerPerceptron ? 
body: In [`multilayer_perceptron.py`](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neural_network/multilayer_perceptron.py) at [line 623](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neural_network/multilayer_perceptron.py#L362) we have
```python
activations = [X]
activations.extend(np.empty((batch_size, n_fan_out)) for n_fan_out in layer_units[1:])
deltas = [np.empty_like(a_layer) for a_layer in activations]
coef_grads = [ np.empty((n_fan_in_, n_fan_out_)) 
                        for n_fan_in_, n_fan_out_ in zip(layer_units[:-1], layer_units[1:]) ]
```
and at [line 268](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neural_network/multilayer_perceptron.py#L268) we have defined
```python
self.n_layers_ = len(layer_units) 
```
as such `activations` and `deltas` should has the same the layer_units. 

However, in the doc strings from other methods of this class, it is claimed at both `activations` and `deltas` has the length of `self.n_layers - 1`, and it seems to me the logic within those methods also follows this assumption. For exampe at [line 236](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neural_network/multilayer_perceptron.py#L236), we have 
```python
last = self.n_layers_ - 2
deltas[last] = activations[-1] - y
```
This does not look right to me. If the network has only three layer (input, hidden, and output) the last would be refer to the the hidden layer. But if my understanding is right the deltas formula is meant for the output layer.

user: Louis-Tian
issue_8978: state: open
title: Vectorize text based on word embeddings
body: I experimented a bit with word embeddings and sentence embeddings recently, and put this this (still a bit rough) repo: https://github.com/jorisvandenbossche/wordembeddings. I am opening an issue here to make people aware of it, and to see if it is worth to discuss whether some things would be interesting for scikit-learn.

Summary of what I did: based on pre-trained word vectors (I used the ones of GloVe, https://nlp.stanford.edu/projects/glove/), I wrote a sklearn compatible Transformer to convert a text record into a numerical feature vector based on those word embeddings (similar as you would do with a CountVectorizer or TfIdfVectorizer). 
I applied this on a few cases based on some SemEval datasets (and comparing with the result of a paper on this, see the overview in the repo).

Some general remarks:

- As opposed to the CountVectorizer or TfIdfVectorizer, this transformer needs external inputs: you need to provide (large) pre-trained vectors (in the current version you need to provide this as a dict), making this less straight-forward to use. For the weighted average you also need to provide general word frequencies (for which I now used the `wordfreq` package).

- The pre-trained word vectors are of course only for words, where you typically have multiple words/sentences. So you need to combined those somehow. The simplest is just to average. I also implemented a weighted average (but drawback: you need to provide general word frequencies) and a weighted average with a reduction based on the singular values from this paper: A Simple but Tough-to-Beat Baseline for Sentence Embeddings, Arora *et al*. 2017, https://openreview.net/pdf?id=SyK00v5xx (but drawback: although this seems to work well, it is a very recent paper and hence not really suited for sklearn)

- For now, the simple cases on which I applied a sklearn model after using the word embeddings transformer don't give very good results (but, I also didn't do much optimization of hyperparameters). In the case of the SICK entailment data, the TfIdfVectorizer actually works as good as the word embeddings ([notebook](https://github.com/jorisvandenbossche/wordembeddings/blob/master/sts_sick_entailment.ipynb)) using the default values of the MLPClassifier for both. 
  It would be also be nice to have a good example dataset, as many of the ones from SemEval that I used just look at similarity between sentences. 

cc @GaelVaroquaux @ogrisel 
user: jorisvandenbossche
issue_8971: state: open
title: Test support for CV splitter where y required in *CV
body: #8426 shows that we can easily miss supporting diverse CV splitters by not correctly passing y (and even other arguments) to .split(...). This is a risk in specialised CV classes and we should ideally have a genetic test to make sure split is passed both X and y at least in all cases.
user: jnothman
issue_8958: state: open
title: Adding a robots.txt to the website
body: Google is indexing old versions of the docs, leading to problems such as #4736.

I suggest adding a robots.txt to fix the problem. I suggest the following content:

<pre>
User-agent: *
Disallow: /*/ 
Allow: /stable
</pre>

I am not sure that the disallow line is correct, though :$. But I think that it is worth trying.

What do people think?
user: GaelVaroquaux
issue_8943: state: open
title: LatentDirichletAllocation  Perplexity too big on Wiki dump
body: #### Description
LDA Perplexity too big on Wiki dump

#### Steps/Code to Reproduce
I used `gensim` `WikiCorpus` to obtain the Bag-of-Words for each document, then vectorised it using `scikit-learn` `CountVectorizer`. All is from the En Wiki dump dated 2017-04-10. I'm using the latest `scikit-learn`.

The LDA used non-informative prior,

```python
lda = LatentDirichletAllocation(
    n_topics=20,
    doc_topic_prior=1.0,
    topic_word_prior=1.0,
    max_iter=1000,
    total_samples=n_docs,
    n_jobs=-1
)

for _ in range(runs):
    lda.partial_fit(training_sample)
    print(np.log(lda.perplexity(test_docs)))
```

#### Expected Results
7.x - 8.x, which is what the Spark LDA would give and a personal code as well. As a baseline, the entropy (log-perplexity) of the mean of all the docs is 8.3 (I manually computed the entropy of M1 of the generated corpus for Python, it gives 8.3; for the corpus generated for Spark, it also gave 8.3). Normally after seeing 20% of data it should print out 8.3 or inferior.

#### Actual Results
13.8 all the way

#### Versions
All Python packages are of the latest versions as of reporting.


user: jli05
issue_8939: state: open
title: NMF is extremely slow.
body: Hi,

I haven't had a chance to look at the sklearn code to figure out where the bottleneck is, but the currently implementation of NMF is vastly slower than it needs to be.  I wanted to factor a very sparse 10,000 x 50,000 matrix and left it running last night to find that it still hadn't converged this morning.  I implemented my own version using the Fast Hierarchical Least Squares algorithm from the Cichocki et. al.  paper referenced on the NMF page with one iteration of regular Alternating Least Squares as initialization, and it converged within a tolerance of 1e-4 in 20 iterations of Fast HALS in less than 5 minutes.  If I had to guess, I think the computation of the error is probably *much* slower than it needs to be, and that may be substantially slowing the algorithm down.  When I first implemented my version, I used: 
```
np.linalg.norm(X-A.dot(B.transpose))
```
which is extremely slow when X is sparse.
```
Y = A.dot(B.transpose())
np.sqrt(X.power(2).sum()+np.sum(np.square(Y))+2*((X*Y).sum()))
```
is orders of magnitude faster since scipy fixed sparse*dense to return sparse.  I would appreciate it if someone is willing to look into this.

Thanks,
Shawn

user: shawnjhenry
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_8934: state: open
title: Feature Request: Partial Least Squares Discriminant Analysis macro
body: Sklearn implements [Partial Least Squares Regression](http://scikit-learn.org/stable/modules/generated/sklearn.cross_decomposition.PLSRegression.html) (PLSR) but a very common use for this algorithm is a slight variation called Partial Least Squares Discriminant Analysis (PLS-DA), which is when y is categorical instead of continuous. You can pretty easily write PLS-DA using the PLSR from sklearn, as documented in [the first answer to this question](https://stackoverflow.com/questions/18390150/pls-da-algorithm-in-python) but it seems silly that everyone who would want to do PLS-DA has to implement this. Other libraries in R, Matlab, etc... provide an interface to PLS-DA directly, and I think sklearn should too. As I imagine it, this would be a very simple method which just does some LabelEncoding and then calls PLSR, but it would save a lot of folks like myself a lot of time. 
user: alexlenail
issue_8908: state: open
title: Sparse binary/multiclass representation
body: In multi-label classification one of the approaches is to transfor the problem to a single-label case of 1+ single-class/multi-class models. In such cases one needs to provide output / target data for the scikit-learn single-label classifier, but only dense array like formats are allowed in binary/multiclass classification.

One of the constraints happens in: `sklearn.utils.multiclass.type_of_target` where the case of binary/multiclass is detected using:

```python
    if (len(np.unique(y)) > 2) or (y.ndim >= 2 and len(y[0]) > 1):
        return 'multiclass' + suffix  # [1, 2, 3] or [[1., 2., 3]] or [[1, 2]]
    else:
        return 'binary'  # [1, 2] or [["a"], ["b"]]
```

adding before this fragment, the following code:
```python
    if issparse(y):
        # the other case is multilabel and is handled earlier, so here y.shape[1] == 1
        # np.unique on y.data does not include 0, so == 1, or empty is binary
        if len(np.unique(y.data)) > 1:
            return 'multiclass'
        else:
            return 'binary'
```

would allow sparse matrices to provide data for single-label problems, but I'm guessing that more changes are needed, due to matrix vs list addressing problems, right? What do you think? I would be willing to work on implementing support for sparse binary/multiclass outputs for at least a couple of classifiers.
user: niedakh
issue_8893: state: open
title: min_impurity_decrease in DecisionTreeRegressor scales with y.std()
body: I'm not sure if the parametrization of ``min_impurity_decrease`` in DecisionTreeClassifier is great.
Impurity is mse, which depends on the scale of y. So scaling y changes the meaning of the parameter. In particular for a tree that seems like odd behavior.

If we rescale y by y.std(), the top node has impurity 1, and the range of ``min_impurity_decrease`` is the same as for classification. Maybe we should look at what other packages are doing, though.
user: amueller
issue_8889: state: open
title: [RFC] mRMR feature selection
body: Speaking with @jorisvandenbossche IRL, we come to discuss about the mRMR feature selection among other methods.

#5372 intended at first to implement the mRMR with mutual information as a metric. However, it has been merged such that mutual information could be used in the `SelectKBest` class. It was also discuss that the mRMR mechanism could be implemented in a separate PR with the possibility to plug any metric.

Therefore, I was wondering if scikit-learn will be interested to have this transformer to perform feature selection.

I would be interested to know the different opinions
@agramfort @MechCoder @jnothman @GaelVaroquaux @amueller @jorisvandenbossche 
user: glemaitre
issue_8884: state: open
title: coclust package has much better results on benchmark datasets
body: It seems that the coclust package works better than our algorithms on some benchmark datasets.
We should investigate this:
http://coclust.readthedocs.io/en/v0.2.1/examples.html

(I'm not sure if the benchmarks are on the website)

user: amueller
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_8853: state: open
title: Different prediction of tree classifier between 32 bits and 64 bits
body: #### Description

While having some test failing in `imbalanced-learn` with a 32 bits machine (we are kinda poor ;)), we got to the point that the `DecisionTreeClassifier` can return different results between a 32 and a 64 architecture.

#### Steps/Code to Reproduce

```python
import numpy as np                                                                                                                                                                                                               
from sklearn.tree import DecisionTreeClassifier                                                                      
                                                                                                                     
RND_SEED = 0                                                                                                         
X = np.array([[-0.3879569, 0.6894251], [-0.09322739, 1.28177189],                                                    
              [-0.77740357, 0.74097941], [0.91542919, -0.65453327],                                                  
              [-0.03852113, 0.40910479], [-0.43877303, 1.07366684],                                                  
              [-0.85795321, 0.82980738], [-0.18430329, 0.52328473],                                                  
              [-0.30126957, -0.66268378], [-0.65571327, 0.42412021],                                                 
              [-0.28305528, 0.30284991], [0.20246714, -0.34727125],                                                  
              [1.06446472, -1.09279772], [0.30543283, -0.02589502],                                                  
              [-0.00717161, 0.00318087]])                                                                            
y = np.array([0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0])                                                          
                                                                                                                     
train_index = [0, 1, 2, 3, 6, 7, 8, 10, 11, 12, 13, 14]                                                              
test_index = [4, 5, 9]                                                                                               
                                                                                                                     
clf = DecisionTreeClassifier(random_state=0)                                                                         
clf.fit(X[train_index], y[train_index])                                                                              
proba = clf.predict_proba(X[test_index])                                                                             
print(proba) 
```

#### Expected Results

This is the result obtained in the 64 bits machine:

```
[[ 1. 0.]
[ 0. 1.]
[ 1. 0.]]
```

#### Actual Results

This is the result obtained for the 32 bits machine:

```
[[ 1. 0.]
[ 0. 1.]
[ 0. 1.]]
```

#### Versions

Ubuntu 16.04 - 32/64 bits
Python 2.7.13
numpy(1.12.1) ; scipy (0.19) ; scikit-learn (0.18.1) installed from conda
user: glemaitre
issue_8849: state: open
title: Run more examples that do not start with plot_ on CircleCI
body: From https://github.com/scikit-learn/scikit-learn/pull/8847#issuecomment-300015905:

> and we should have a CI test for non-plotted examples or convert as many as possible to plots

My proposal is to have a convention like `run_` for examples that do not produce any plots. sphinx-gallery allows to have a regex to specify which examples you want to run. It could be something like `plot_|run_`. See the [doc](http://sphinx-gallery.readthedocs.io/en/latest/advanced_configuration.html?highlight=filename_pattern#building-examples-matching-a-pattern) for more details.

I looked at the examples whose filename is not starting with `plot_`. Timings are in seconds and in increasing order.

```
examples/feature_selection/feature_selection_pipeline.py 1.39
examples/exercises/digits_classification_exercise.py 1.47
examples/applications/svm_gui.py 1.86
examples/missing_values.py 2.01
examples/model_selection/randomized_search.py 2.02
examples/feature_stacker.py 2.14
examples/text/document_clustering.py 3.21
examples/linear_model/lasso_dense_vs_sparse_data.py 3.98
examples/text/hashing_vs_dict_vectorizer.py 4.78
examples/model_selection/grid_search_digits.py 8.29
examples/text/document_classification_20newsgroups.py 8.93
examples/applications/topics_extraction_with_nmf_lda.py 10.53
examples/applications/face_recognition.py 25.02
examples/bicluster/bicluster_newsgroups.py 25.72
examples/hetero_feature_union.py 116.22
examples/applications/wikipedia_principal_eigenvector.py 139.77
examples/model_selection/grid_search_text_feature_extraction.py 156.86
```

With this in mind I would be in favour of running all the examples but `svm_gui.py` and the last three examples.

More details: 
`svm_gui.py` pops up a gui so it should probably not be run. Whether we should run `wikipedia_principal_eigenvector.py` and `grid_search_text_feature_extraction.py` which each takes more than 2 minutes is up for debate. On top of that, some of them may require data download that is not using the typical `~/scikit_learn_data` (e.g. the Wikipedia one). If that is the case these examples would not benefit from the CircleCI cache.

user: lesteve
issue_8834: state: open
title: Reduce memory usage for spectral clustering
body: This is not a bug per se, but rather a feature request.

I'm currently dealing with memory issues.
The size of my affinity matrix is around 32768 x 32768, which takes about 8GB of memory. Having 32GB of RAM should be more than enough. Except it's not.

The spectral embedding function compute the laplacian matrix, which is the same size as the affinity matrix. This temporarily doubles the memory usage. The other 16GB are probably taken by the BLAS computing the eigenvectors, so I guess not much can be done about that.

My proposition would be to add an argument to the `graph_laplacian` function to ask to overwrite the input matrix instead of allocating a new one. And add the same argument all the way up to the `SpectralClustering` class.
user: Celelibi
issue_8823: state: open
title: Question about the computation of the lower bound in mixture.bayesian_mixture.py
body: In method _compute_lower_bound of mixture.bayesian_mixture.py all the terms that should be in the lower bound aren't there.
That's normal for constant terms but for the others is there an explanation ?

(The formula in Bishop book is : 

![lower_bound_formula](https://cloud.githubusercontent.com/assets/9931663/25631072/93412cb8-2f6f-11e7-938e-3ec3ce2dc0ab.png)
)
user: 14thibea
issue_8809: state: open
title: Confusing error message in fit() methods using SVD
body: I hope the code says all:

```python
>>> from sklearn.cluster import SpectralCoclustering, SpectralBiclustering
>>> import numpy as np
>>> a = np.array([[1,0],[0,0]])
>>> SpectralCoclustering(2).fit(a)
```
results in the following traceback:
```python-traceback
/usr/lib/python3.6/site-packages/sklearn/cluster/bicluster.py:38: RuntimeWarning: divide by zero encountered in true_divide
  row_diag = np.asarray(1.0 / np.sqrt(X.sum(axis=1))).squeeze()
/usr/lib/python3.6/site-packages/sklearn/cluster/bicluster.py:39: RuntimeWarning: divide by zero encountered in true_divide
  col_diag = np.asarray(1.0 / np.sqrt(X.sum(axis=0))).squeeze()
/usr/lib/python3.6/site-packages/sklearn/cluster/bicluster.py:48: RuntimeWarning: invalid value encountered in multiply
  an = row_diag[:, np.newaxis] * X * col_diag
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/sklearn/cluster/bicluster.py", line 123, in fit
    self._fit(X)
  File "/usr/lib/python3.6/site-packages/sklearn/cluster/bicluster.py", line 281, in _fit
    u, v = self._svd(normalized_data, n_sv, n_discard=1)
  File "/usr/lib/python3.6/site-packages/sklearn/cluster/bicluster.py", line 137, in _svd
    **kwargs)
  File "/usr/lib/python3.6/site-packages/sklearn/utils/extmath.py", line 364, in randomized_svd
    power_iteration_normalizer, random_state)
  File "/usr/lib/python3.6/site-packages/sklearn/utils/extmath.py", line 258, in randomized_range_finder
    Q, _ = linalg.lu(safe_sparse_dot(A, Q), permute_l=True)
  File "/usr/lib/python3.6/site-packages/scipy/linalg/decomp_lu.py", line 178, in lu
    a1 = asarray_chkfinite(a)
  File "/usr/lib/python3.6/site-packages/numpy/lib/function_base.py", line 1215, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs
```
The following gives the same error then:
```python
>>> SpectralBiclustering(2).fit(a)
```

The array `a` obviously does not contain infs or NaNs but the method complains anyway. I think the error is actually caused by the array `a` being too small, but that's something very different. If that is the case, the error message should be made more relevant, otherwise it should be fixed somehow.
user: jachymb
issue_8806: state: open
title: Maximum recursion depth reached while pickling Birch clustering estimators
body: Because of the highly recursive structure used in Birch clustering, the maximum recursion depth can be easily reached while pickling this estimator.

Here is minimal example,
```py
import numpy as np
from sklearn.cluster import Birch
from sklearn.externals import joblib

np.random.seed(42)

X = np.random.randn(10000, 300)

b = Birch(n_clusters=None, compute_labels=False)
b.fit(X)

joblib.dump(b, '/tmp/birch_obj.pkl')
```

which produces the following traceback (using Linux, Python 3.6, scikit-learn 0.18.1),
<details>
<p>

```pytb
Traceback (most recent call last):
  File "example_birch_recursion_2.py", line 12, in <module>
    joblib.dump(b, '/tmp/birch_obj.pkl')
  File "/home/rth/.miniconda3/envs/freediscovery-env/lib/python3.6/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 483, in dump
    NumpyPickler(f, protocol=protocol).dump(value)
  File "/home/rth/.miniconda3/envs/freediscovery-env/lib/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/home/rth/.miniconda3/envs/freediscovery-env/lib/python3.6/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 280, in save
    return Pickler.save(self, obj)
  File "/home/rth/.miniconda3/envs/freediscovery-env/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/rth/.miniconda3/envs/freediscovery-env/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/home/rth/.miniconda3/envs/freediscovery-env/lib/python3.6/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 280, in save
    return Pickler.save(self, obj)
  File "/home/rth/.miniconda3/envs/freediscovery-env/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/rth/.miniconda3/envs/freediscovery-env/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/home/rth/.miniconda3/envs/freediscovery-env/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
[...]
  File "/home/rth/.miniconda3/envs/freediscovery-env/lib/python3.6/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 280, in save
    return Pickler.save(self, obj)
  File "/home/rth/.miniconda3/envs/freediscovery-env/lib/python3.6/pickle.py", line 469, in save
    self.write(self.get(x[0]))
  File "/home/rth/.miniconda3/envs/freediscovery-env/lib/python3.6/pickle.py", line 450, in get
    if i < 256:
RecursionError: maximum recursion depth exceeded in comparison
```
</p>
</details>

this is mostly due to the inherent pickle limitations and also occurs when using the Python 3.6 pickle instead of `joblib.dump` (also related https://github.com/joblib/joblib/issues/224).

### Additional details / workarounds

This can be fixed by increasing the recursion limit with `sys.setrecursionlimit` however this has it's own limitations, and the main issue is that given some dataset and Birch clustering parameters, there is no guarantee that the fitted estimator will be picklable, even with after increasing the recursion limit to some high value.

For instance, the above example produces a fairly shallow hierarchical tree,
```py
def _compute_htree_size(node, depth=0, cluster_id=0):
    max_depth = depth
    for el in node.subclusters_:
        if el.child_ is not None:
            cluster_id += 1
            subcluster_depth, cluster_id = _compute_htree_size(
                     el.child_, depth=depth+1, cluster_id=cluster_id)
            max_depth = max(depth, subcluster_depth)
    return max_depth, cluster_id

max_htree_depth, n_htree_subclusters = _compute_htree_size(b.root_)
print('Input dataset size: n_samples={}, n_features={}'.format(*X_lsa.shape))
print('Max depth of the hierarchical tree: ', max_htree_depth)
print('Number of subclusters:', n_htree_subclusters)
```
which yields,
```
Input dataset size: n_samples=10000, n_features=300
Max depth of the hierarchical tree:  2
Number of subclusters: 389
```
and yet it is sufficient to reach the default maximum recursion depth of 1000.  Increasing the maximum recursion limit to `10000`, fixes the original example, but would break if we set `branching_factor=10` (from the default `50`) in `Birch` (which produces a still reasonable hierarchical tree 7 levels deep and  with 3478 subclusters)...

More specifically, the issue is with pickling the `Birch.root_` attribute which consists of recursively nested  `_CFNode` / `_CFSubcluster` objects.

Not sure if there is a better solution than converting the `Birch.root_` to a less recursive object when writing to disk (e.g. a list linked with some numerical `subcluster_id`) then converting it back to a nested structure when reading from disk?
user: rth
issue_8755: state: open
title: train_test_split fails for too many values (32bit only)
body: Consider the following code:

```py
import numpy as np
from sklearn.model_selection import train_test_split

n = 10000
y = np.random.randint(0, 2, size=n)

y_train, y_test = train_test_split(y, train_size=int(n/2),
                                   test_size=int(n/2), stratify=y, random_state=123)

print('num train: {}'.format(len(y_train)))
print('train mean: {}'.format(y_train.mean()))
print('num test: {}'.format(len(y_test)))
print('test mean: {}'.format(y_test.mean()))
```

When n=10,000, I correctly obtain:
```py
num train: 5000
train mean: 0.4958
num test: 5000
test mean: 0.4958
```

But for larger n, such as n=100,000, I get the following error:
```pytb
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/home/scott/Development/scratch/sklearn/stratify.py in <module>()
      6 y = np.random.randint(0, 2, size=n)
      7 
----> 8 y_train, y_test = train_test_split(y, train_size=n/2, test_size=n/2, stratify=y, random_state=123)
      9 
     10 print 'num train: {}'.format(len(y_train))

/home/scott/anaconda/lib/python2.7/site-packages/sklearn/model_selection/_split.pyc in train_test_split(*arrays, **options)
   1700     train, test = next(cv.split(X=arrays[0], y=stratify))
   1701     return list(chain.from_iterable((safe_indexing(a, train),
-> 1702                                      safe_indexing(a, test)) for a in arrays))
   1703 
   1704 

/home/scott/anaconda/lib/python2.7/site-packages/sklearn/model_selection/_split.pyc in <genexpr>((a,))
   1700     train, test = next(cv.split(X=arrays[0], y=stratify))
   1701     return list(chain.from_iterable((safe_indexing(a, train),
-> 1702                                      safe_indexing(a, test)) for a in arrays))
   1703 
   1704 

/home/scott/anaconda/lib/python2.7/site-packages/sklearn/utils/__init__.pyc in safe_indexing(X, indices)
    110             return X.take(indices, axis=0)
    111         else:
--> 112             return X[indices]
    113     else:
    114         return [X[idx] for idx in indices]

IndexError: arrays used as indices must be of integer (or boolean) type
```

And for n=1,000,000, I don't get an exception, instead the strange results:
```py
num train: 1785
train mean: 0.414565826331
num test: 894
test mean: 0.414988814318
```

Why is this? Is this a bug? Does train_test_split fail with too many values?
user: scottmsul
issue_8740: state: open
title: Spectral clustering for large matrix
body: My affinity  matrix size is 57600 by 57600. But when I use spectral clustering for that, the function throws 'memory error'.

Why does the function throws 'memory error'? Is it because of complexity of algorithm ?

My machine has 8 Gb ram ,but no GPU.
user: gunjannaik
issue_8737: state: open
title: LogisticRegressionCV doesn't calculate score as a multiclass problem
body: ```python
from sklearn.linear_model import LogisticRegressionCV
from sklearn import datasets
iris = datasets.load_iris()

LogisticRegressionCV(scoring="f1").fit(iris.data, iris.target)
```
This runs and I think it should error.
By default ``f1`` uses binary averaging, which makes no sense for multiclass IIRC.
I'm sure @jnothman knows more about this than me.
I'd argue this is a bug in the ``f1_score`` which should error if ``average='binary'`` but the targets are not binary. 
Though I vaguely remember that we discussed that we want to enable this behavior to compute the f1-score for a single class that is selected by ``pos_label`` but that seems to be covered by 
this
> The class to report if average='binary' and the data is binary. If the data are multiclass or multilabel, this will be ignored; setting labels=[pos_label] and average != 'binary' will report scores for that label only.

The docstring seems exactly right to me but the behavior is different.


user: amueller
issue_8730: state: open
title: MultiOutputClassifier doesn't support sparse y
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
MultiOutputClassifier doesn't train on RCV1
The error is 
```Found input variables with inconsistent numbers of samples: [804414, 1]```

#### Steps/Code to Reproduce
```py
from sklearn.datasets import fetch_rcv1
data_location = '../../../Data';
rcv1 = fetch_rcv1(data_home = data_location)

from sklearn.multioutput import MultiOutputClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import SGDClassifier
clf_one = SGDClassifier(loss='log', penalty='l1', alpha=1e-4, n_iter=30, random_state=42, warm_start=True)
clf = MultiOutputClassifier(clf_one)

clf = clf.fit(rcv1.data, rcv1.target)
```

#### Expected Results
Expected this to work

#### Actual Results
```pytb
C:\Users\fedor\AppData\Local\Continuum\Anaconda2\lib\site-packages\sklearn\utils\validation.py:531: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)

C:\Users\fedor\AppData\Local\Continuum\Anaconda2\lib\site-packages\sklearn\utils\validation.pyc in check_consistent_length(*arrays)
```

#### Versions
```
Windows-10-10.0.14393
('Python', '2.7.12 |Anaconda custom (64-bit)| (default, Jun 29 2016, 11:07:13) [MSC v.1500 64 bit (AMD64)]')
('NumPy', '1.12.1')
('SciPy', '0.19.0')
('Scikit-Learn', '0.18.1')
```

<!-- Thanks for contributing! -->

user: fedorzh
issue_8722: state: open
title: manifold.MDS effect of n_jobs is unclear
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
When I run manifold.MDS on a 1024*1024 dissimilarity matrix with `n_jobs=1`, it seems to use all 8 cores, but most of the time is spent in the kernel. Which is quite surprising.

#### Steps/Code to Reproduce

    mat = np.random.rand(1024, 1024)
    mat = mat + mat.T

    mds = skmanifold.MDS(dissimilarity='precomputed', n_jobs=1)
    mds.fit(mat)

#### Expected Results
Multidimensional scaling seems to be a rather compute-intensive task, so I wouldn't expect the computation to spend most of its time in the kernel.

I'd also expect to only get `n_jobs` threads or processes doing the actual computation, not 8 times more.

Or, if this is a non-issue, maybe I'd expect the documentation to say what this argument does.

#### Actual Results
The example code above seems to create 7 additional threads which use all 8 cores that spend most of their time in the kernel.

With `n_jobs=2`, the main process creates 3 threads and forks 2 new processes, each having 8 threads.

I am not sure this is of any use to say it, but the `n_jobs` argument seems to have absolutely no effect on the processing time. Here is the processing time with respect to the value of `n_jobs`.

    # n_jobs seconds
    1 24.469351
    2 24.655377
    3 25.569757
    4 24.862707
    5 25.483622
    6 26.072145
    7 26.072289
    8 25.579232


#### Versions
Linux-4.9.0-1-amd64-x86_64-with-debian-9.0
Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170118]
NumPy 1.12.0
SciPy 0.18.1
Scikit-Learn 0.18
user: Celelibi
issue_8714: state: open
title: [WIP] Online Inference for BayesianGaussianMixture
body: It would be great if someone implemented partial_fit for BayesianGaussianMixture using mean field [stochastic variational inference.](http://jmlr.org/papers/volume14/hoffman13a/hoffman13a.pdf)  Fitting with EM is way too slow to use for online clustering on large datasets, and right now the only online clustering method in sklearn is MiniBatchKMeans, which doesn't handle noisy data very well.  Alternatively, a [parallel Gibbs sampler](https://people.csail.mit.edu/mattjj/papers/nips2013hogwild.pdf) would work too but is likely a lot more work.
user: jengelman
issue_8702: state: open
title: TruncatedSVD explain_variance_ratio going up?
body: Using Normalizer and TruncatedSVD I get this:
```
array([ 0.015,  0.036,  0.013,  0.008,  0.007,  0.007,  0.006,  0.005,
        0.005,  0.005,  0.005,  0.004,  0.004,  0.004,  0.004,  0.003,
        0.003,  0.003,  0.003,  0.003,  0.003,  0.003,  0.003,  0.003,
        0.003,  0.003,  0.003,  0.003,  0.003,  0.002,  0.002,  0.002,
        0.002,  0.002,  0.002,  0.002,  0.002,  0.002,  0.002,  0.002,
        0.002,  0.002,  0.002,  0.002,  0.002,  0.002,  0.002,  0.002,
        0.002,  0.002,  0.002,  0.002,  0.002,  0.002,  0.002,  0.002,
        0.002,  0.002,  0.002,  0.002,  0.002,  0.002,  0.001,  0.001,
        0.001,  0.001,  0.001,  0.001,  0.001,  0.001,  0.001,  0.001,
        0.001,  0.001,  0.001,  0.001,  0.001,  0.001,  0.001,  0.001,
        0.001,  0.001,  0.001,  0.001,  0.001,  0.001,  0.001,  0.001,
        0.001,  0.001,  0.001,  0.001,  0.001,  0.001,  0.001,  0.001,
        0.001,  0.001,  0.001,  0.001])
```
That seems odd to me. This is running CountVectorizer on the aclIMDB data. Shouldn't they be sorted by eigenvalues?
user: amueller
issue_8663: state: open
title: [MRG] Calibration curve weighted
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
<!-- Example: Fixes #1234 -->


#### What does this implement/fix? Explain your changes.

This pull request aims to add the `sample_weight` parameter to the `calibration_curve` method in order for it to behave similarly to `cross_val_score`, `classification_report`, `roc_auc_score`, `roc_curve`, `average_precision_score`, `precision_recall_curve`, `brier_score_loss`, `precision_score`, `recall_score`, and `f1_score`.

Additionally, this implementation checks to make sure that there is a positive weighted number of samples. I am uncertain of whether this last requirement is strictly necessary.

```python
if sample_weight.sum() <= 0:
```

#### Any other comments?

In particle physics, it is common for Monte Carlo simulations to incorrectly generate the number of background and signal events by a small amount when producing a sample. As a means to remedy the composition of the sample a signal event may receive, for example, a weight of 1.3, while a background event may be given a weight of 0.8. Because the `calibration_curve` method is performing a particular frequency count on the prediction values of the sample it is important that the sample composition be corrected on an event-by-event basis. This is the physics motivation behind my PR. Thank you for your considerations. 

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: ecampana
issue_8655: state: open
title: The  haversine metric in DBSCAN  is too slow
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
The  haversine metric in the DBSCAN is too slow, it could be much faster using 
the 'cosine' distance for cartesian coordinates of the unit sphere.
```
from scipy.spatial.distance import  cdist
dist= np.arccos(-cdist(pos_ref,pos,'cosine')+1)
dist [np.isnan(dist)] = 0
```
where pos and pos_ref are the (x,y,z) cartesian coordinates on the uni sphere for the reference 
point  the array of points. With this implementation you get rid of all the trigonometric functions used 
in the  haversine formula (you would have just one), and you need just to transform (longitude,latitude)  to (x,y,z) once.
From my tests you should get an improvement in terms of speed of a factor of ~ 20.
Use the following script to test.
  
```
from  scipy.spatial.distance import cdist
from  sklearn.neighbors import DistanceMetric
import timeit
import numpy as np


def spherical_to_cartesian(lon,lat, ind=None,unpacked=False):
   
    
    ang_coord=np.column_stack((lon,lat))
    x=np.cos(ang_coord[:,1])*np.cos(ang_coord[:,0])
    y=np.cos(ang_coord[:,1])*np.sin(ang_coord[:,0])
    z=np.sin(ang_coord[:,1])

    return np.column_stack((x,y,z))



def d_sp(x1,x):
    
    return np.arccos(-1.0*cdist(x1,x,'cosine')+1.0)
    
def d_sk(dist,x1,x):
    
    return dist.pairwise(x1,x)

N=100000
ra=np.random.uniform(-1,1,N)*np.pi
dec=np.random.uniform(-0.5,0.5,N)*np.pi
r=np.ones(N)

X=spherical_to_cartesian(ra,dec)
X1=spherical_to_cartesian(np.pi*0.25,np.pi*0.25)

print timeit.timeit("d_sp(X1,X)", setup="from __main__ import d_sp,X1,X",number=10)

dist = DistanceMetric.get_metric('haversine')

print timeit.timeit("d_sk(dist,[[np.pi*0.25,np.pi*0.25]],np.column_stack((dec,ra)))", setup="from __main__ import d_sk,dist,np,dec,ra",number=10)

```

user: andreatramacere
issue_8651: state: open
title: Add plotting function for silhouette plot
body: There's a nice example for silhouette plots:
http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html#sphx-glr-auto-examples-cluster-plot-kmeans-silhouette-analysis-py

I think this would be a good candidate for getting it's own plotting function.
user: amueller
issue_8647: state: open
title: calinski_harabaz_score not used in any example, unclear when to use
body: The docs on calinski_harabaz_score are really thin and it's not used in any example.
It's very unclear how it relates to and compares against silhouette score, which is the only other unsupervised cluster metric.

[Also, guess what I'm working on today]
user: amueller
issue_8646: state: open
title: Cluster evaluation user guide should be split in supervised and unsupervised, possibly with a table summarizing them.
body: The beginning of the cluster evaluation metric section should list all the metrics and list whether they are supervised or unsupervised, similarly to how listings are done for the other metrics.
Maybe this would be a chance to actually move this section to the metrics page in the user guide where this belongs? [Had we decided against doing that, don't remember any more]
user: amueller
issue_8642: state: open
title: Implement `nbytes` or `__sizeof__`  (or equivalent) for estimators
body: In `dask.distributed` we use a [dispatch on type](https://github.com/dask/distributed/blob/master/distributed/sizeof.py#L14) to determine memory overhead for intermediate results. Having a rough sense of the size of intermediate results is useful for the scheduler as it often correlates to the cost of serializing that intermediate between workers.

The default is to fallback to `sys.getsizeof` which calls the [`__sizeof__` method](https://docs.python.org/3/library/sys.html#sys.getsizeof) on the object. It would be useful if this (or an equivalent method) was implemented for scikit-learn estimators.

A naive generic implementation for estimators might be:

```python
def __sizeof__(self):
    return sum(x.nbytes if hasattr(x, 'nbytes') else getsizeof(x)
               for x in self.__dict__.values())
```

It'd probably even be fine to ignore (or approximate) the memory usage of parameters, and just focus on the memory usage of the results of `fit`. This may be straightforward for numpy arrays, but less clear for things like [trees](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/tree/_tree.pyx#L492).
        
user: jcrist
issue_8635: state: open
title:  Pull out the _changed.html logic for Circle to see the impacted files locally
body: For my last PR #8556, I used the circle functionality for showing my changes. However, this was done after the commit was pushed so if I have detected something I had to add another commit. 

Thus, @jnothman proposed that a script launched locally could be used to reproduce the _changed.html file locally and so anticipate the verification.

Let me know if it is worth doing it and how. In a next PR, I will show a functional script.

What the doc will say about the functionality (not present in 0.18 but on master):
```rst
You may also be asked to show your changes when it's built. When you create
a pull request or make changes in an existing one modifying the docs, CircleCI
automatically builds them. Thus, you can easily view your changes in the built
artifacts using the following formula:

``http://scikit-learn.org/circle?{BUILD_NUMBER}``

We attempt to assemble a more precise set of changed files in the
documentation at:

``http://scikit-learn.org/circle?{BUILD_NUMBER}/_changed.html``

Note: When you visit the details page of the CircleCI tests, you can find your
BUILD_NUMBER mentioned as 'build #' which is different from your pull request
number, which is presented as 'pull/#'.

```
user: Morikko
issue_8632: state: open
title: Surprising result in BayesianGaussianMixture
body: I'm surprised by the results of BayesianGaussianMixture
```python
from sklearn.mixture import BayesianGaussianMixture
from sklearn.datasets import make_blobs
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

rng = np.random.RandomState(3)
X, y = make_blobs(n_samples=500, centers=10, random_state=rng, cluster_std=[rng.gamma(1.5) for i in range(10)])

fig, axes = plt.subplots(2, 4)
gammas = np.logspace(-5, 5, 4)
for gamma, ax in zip(gammas, axes.T):
    bgmm = BayesianGaussianMixture(n_components=10, weight_concentration_prior=gamma).fit(X)
    ax[0].scatter(X[:, 0], X[:, 1], s=5, alpha=.6, c=plt.cm.Vega10(bgmm.predict(X)))
    ax[0].set_title("gamma={:.2f}".format(gamma))
    ax[1].bar(range(10), bgmm.weights_)
```
![image](https://cloud.githubusercontent.com/assets/449558/24217565/b050eca4-0f16-11e7-9806-ec85f9154ff3.png)

I'm changing gamma pretty drastically but the distribution of weights doesn't seem to change. Is that expected?
user: amueller
issue_8628: state: open
title: UnaryEncoder
body: I'm sure we've discussed this before, but I'm not sure where, and there certainly does not appear to be an active PR. For ordinal (and discretized; see #7668) features, a "unary" encoding (is there a better name for this) is more informative than a one-hot encoding. For k values 0, ..., k - 1 of the ordinal feature x, this creates `k - 1` binary features such that the `i`th is active if `x > i (for i = 0, ... k - 1)`. Below is an initial implementation.

```python
class UnaryEncoder(BaseEstimator, TransformerMixin):
    def __init__(self, n_values):
        self.n_values = n_values

    def fit(self, X, y=None):
        return self

    def transform(self, X):
        values = np.arange(self.n_values - 1)
        X = check_array(X)
        Xt = np.hstack([values < X[:, i, None] for i in range(X.shape[1])])
        return Xt
```

```python
>>> UnaryEncoder(3).fit_transform([[0], [1], [2]])
array([[0, 0],
       [ 1, 0],
       [ 1,  1]])
```
user: jnothman
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_8614: state: open
title: Add wrapper class that changes threshold value for predict
body: This was discussed before, but not sure if there's an issue.
We should have a wrapper class that changes the decision threshold based on a cross-validation (or hold-out, pre-fit) estimate.
This is very very common so I think we should have a built-in solution.
Simple rules for selecting a new threshold are:
- picking the point on the roc curve that's closest to the ideal corner
- picking the point on the precision-recall curve that's closest to the ideal corner
- optimize one metric while holding another one constant: find the best threshold to yield the best recall with a precision of at least 10% for example. We could also make this slightly less flexible and just say "the median of the largest threshold over cross-validation that yields at least X precision".
user: amueller
issue_8604: state: open
title: [MRG] Make BaseSVC.classes_ sample_weight aware (#8566)
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
Fixes #8566 


#### What does this implement/fix? Explain your changes.
When sample_weights are such that some classes do not have any assigned weight, libsvm results exclude the missing classes. This causes BaseSVC.classes_ to be out of sync with libsvm results. This commit fixes the issue by limiting BaseSVC.classes_ to the ones that have a sample_weight.

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: satyan-g
labels: Needs Decision: Requires decision
issue_8602: state: open
title: FEAT Large Margin Nearest Neighbor implementation
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
<!-- Example: Fixes #1234 -->


#### What does this implement/fix? Explain your changes.
Hi! This is an implementation of the Large Margin Nearest Neighbor algorithm that follows the original implementation of the author's (K.Weinberger) Matlab code (see also my docstrings). As far as I know there is no prior implementation of this version of the algorithm in python and no version at all in scikit-learn. The implementation lies in a single file (sklearn/neighbors/lmnn.py). Maybe it should go directly in the sklearn/neighbors/classification.py but I thought it would make things too cluttered. It is basically a subclass of the `KNeighborsClassifier` with a lot of added functionality. I put also an example in examples/neighbors/plot_lmnn_classification.py that is basically a copy of the simple nearest neighbor example in examples/neighbors/plot_classification.py. 

#### Any other comments?
I run all scikit-learn tests I could find and it now passes all of them. There is still a TODO in the code which I think is a minor issue (the algorithm works anyway). Somehow during fitting, I need to pass a possibly changed attribute (n_neighbors) to the superclass and call the fit method of the superclass. I would appreciate any comments and/or help to finalize this. Thanks!

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: johny-c
labels: Move to scikit-learn-extra: This PR should be moved to the scikit-learn-extras repository
Needs Decision - Close: Requires decision for closing
issue_8590: state: open
title: Feature request: forcing a Gram matrix to be positive definite
body: 

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
Kernel methods like SVR use a similarity measure rather than a vector of features. The prior knowledge is encoded by making a domain-appropriate similarity measure. However, there is a technical requirement that is sometimes hard to meet: a valid kernel must be symmetric, and positive semidefinite. 

#### Suggested solution
In some circumstances, it works well to coerce the Gram matrix into a symmetric matrix by averaging it with its transpose; and then coercing it into a PSD matrix by discarding negative eigenvalues. For R users, the latter operation is available here: https://stat.ethz.ch/R-manual/R-devel/library/Matrix/html/nearPD.html.

The user can do these calculation for SVR, which supports precomputed kernels, but not for the other kernel methods in sklearn, which do not.

Even for SVR, the second transformation is more efficient if it is integrated into the Cholesky decomposition which is already performed by the sklearn package.

I therefore propose an option coerce_kernel=False, on the fit method for SVR.
user: chrishmorris
issue_8573: state: open
title: BUG: Inconsistent behavior when passing a Pipeline to BaggingClassifier
body: Let's run BaggingClassifier on a base estimator that accepts sample_weight. 

```python
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.svm import SVC
from sklearn.ensemble import BaggingClassifier
#------------------------------------------------------------------------------
def main():
    #1) prepare data
    np.random.seed(0)
    feat=np.random.normal(size=(1000,2))
    lbl=np.random.choice(a=2,p=[.5,.5],size=1000,replace=True)
    wghts=np.ones(shape=(1000,))
    
    #2) fit using standard SVC
    bg0=BaggingClassifier(base_estimator=SVC(probability=True,random_state=0),n_estimators=100,max_samples=1.,n_jobs=-1,random_state=0)
    bg0=bg0.fit(feat,lbl)
    prob0=bg0.predict_proba(feat)
    #6.1) fit using standard pipe

    #3) fit using a Pipeline that contains only a SVC
    bg1=BaggingClassifier(base_estimator=Pipeline([('clf',SVC(probability=True,random_state=0))]),n_estimators=100,max_samples=1.,n_jobs=-1,random_state=0)
    bg1=bg1.fit(feat,lbl)
    prob1=bg1.predict_proba(feat)
    print 'match?',np.allclose(prob0,prob1)
#------------------------------------------------------------------------------
if __name__=='__main__':main()
```

prob0 and prob1 are very different, even though both BaggingClassifier are intrinsically the same. The reason is, the SVC accepts sample_weight, and the Pipeline does not accept sample_weight.

Finally, this breaks for no good reason.

```python
bg0=bg0.fit(feat,lbl,wghts) # works!
bg1=bg1.fit(feat,lbl,wghts) # breaks...
ValueError: The base estimator doesn't support sample weight
```

numpy version 1.11.3
sklearn version 0.18.1
user: JohnNapier
issue_8563: state: open
title: [MRG] add random_state in tests estimators
body: 
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
Fix #8194 

#### What does this implement/fix? Explain your changes.
Pass random_state in all test cases where it was missing in the estimator's invocation. 

#### Any other comments?
Added only in a few tests yet, nosetests ran OK. If it is correct and there are no further problems, I will add all the rest. Also, should I add in the contributing docs that when possible all estimators in new tests must be invoked with random_state set?



<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: kkatrio
labels: Needs Decision: Requires decision
issue_8559: state: open
title: BUG: BaggingClassifier.fit(X,y,sample_weight) raises Exception when Pipeline's steps do not expect sample_weight
body: Consider the following pipeline. One step involves a scaler, another a SVC.

```python
import numpy as np
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import BaggingClassifier

# get data
feat=np.random.normal(size=(100,2))
lbl=np.random.binomial(1,.5,size=(100,))
wghts=np.ones(shape=(100,))

# define pipeline
pipe_svc=Pipeline([('scl',StandardScaler()),
                   ('clf',SVC(probability=True))])
```

We can fit this pipeline while passing sample_weight to the SVC.

```python
# fit one
fit_params={'clf__sample_weight':wghts}
pipe_svc.fit(feat,lbl,**fit_params)
```

So far so good. Now, we would like to do the same on an ensemble.

```python
# fit ensemble
bc=BaggingClassifier(base_estimator=pipe_svc,n_estimators=10,max_samples=1.)
bc.fit(feat,lbl,sample_weight=wghts)
```

The problem is, sample_weight is passed to all steps of pipe_svc, and 'scl' is not expecting it. The documentation says that BaggingClassifier can handle pipelines. Given the above, I'm not sure that's the case. One solution would be to allow bc.fit(feat,lbl,**fit_params), so that BaggingClassifier can handle pipelines.

numpy version 1.11.3
sklearn version 0.18.1

Thanks!
user: JohnNapier
issue_8547: state: open
title: [MRG] ENH: multi-output support for BaggingRegressor
body: Closes #8993, #3449, #4848. This minimalistic patch adds multi-output support to BaggingRegressor.


user: dohmatob
issue_8480: state: open
title: Feature selection for categorical variables
body: Currently feature selection on categorical variables is hard.
Using one-hot encoding we can select some of the categories, but we can't easily remove a whole variable.
I think this is something we should be able to do.

One way would be to have feature selection methods that are aware of categorical variables - I guess ``SelectFromModel(RandomForestClassifier())`` would do that after we add categorical variable support.

We should have some more simple test-based methods, though.
Maybe ``f_regression`` and ``f_classif`` could be extended so that they can take categorical features into account?

Alternatively we could try to remove groups of features that correspond to the same original variable. That seems theoretically possible if we add feature name support, but using feature names for this would be putting semantics in the names which I don't entirely like. Doing it "properly" would require adding meta-data about the features, similarly to ``sample_props``, only on the other axis. That seems pretty heavy-handed though.
user: amueller
issue_8474: state: open
title: ENH add support to missing values in NMF
body: Fixes #8447
Fixes #25229

Here I update NMF's multiplicative update solver to handle missing values (`np.nan`).
The solver simply doesn't take into account the loss at missing values, and optimizes the factorization on the rest of the data. This can be useful for imputation, recommendation (a bit out of scope for sklearn), or cross-validation.

~~**Not sure this is in scikit-learn scope though.**~~
___
The solver update is simple. Except for beta = 1, both denominator and numerator (in the multiplicative update) have either `X` or `W * H`, so we can mask them and sum the denominator and the numerator over valid elements. This is referred in the literature as "weighted NMF".

For the coordinate descent solver, the method uses the following factorization, to avoid recomputing `W * H` for each coordinate update of W: `(X - W * H) * H.T = X * H.T - W * (H * H.T)`
During the entire W-update, `X * H.T` and `H * H.T` do not change, so the optimization is fast.
But if we want to handle missing value, `H * H.T` changes for each line of W, which increases heavily the computations.
So I did not update the coordinate descent solver to handle missing values.

For the multiplicative update solver, the approximation `W * H` is used for the update of all coordinates, so we can compute it only once for each update of `W`. This is still more costly than `W * (H * H.T)` as used with beta = 2 without missing values, but this is already what we do for the other beta losses (!= 2).

I skipped the sparse case for now.
___
TODO:
- [ ] add more tests
- [ ] add some documentation and an example
- [ ] add to https://scikit-learn.org/stable/modules/impute.html
user: TomDLT
issue_8471: state: open
title: [MRG] Add Gap safe screening to precomputed ElasticNet and MultiTask
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
<!-- Example: Fixes #1234 -->

#### What does this implement/fix? Explain your changes.
This PR follows #7853 to add screening to precompute ElasticNet and MultiTaskElasticNet.

#### Any other comments?
Needs #7853 to be merged :).


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: tguillemot
issue_8466: state: open
title: Gabor kernel for gaussian process
body: Gabor kernel is missing in the gaussian process package. The covariance function is parameterized as: `k(x, x') = exp(-||x - x'||^2 / (2*l^2)) cos(2*pi*||x - x'|| / p)` in which the hyperparameters are:

- `l`: length scale
- `p`: period

I am happy to work on it if needed.
user: changlan
issue_8451: state: open
title: Unable to use voting classifier when the prediction results contain a negative class(number)
body: When classifiers for outlier detection are used in the Voting Classifier(voting type-hard), the following error occurs.

```
Traceback (most recent call last):
  File "voting.py", line 58, in <module>
    prediction_results = clf.predict(X_test)
  File "/home/tak/anaconda3/lib/python3.5/site-packages/sklearn/ensemble/voting_classifier.py", line 194, in predict
    arr=predictions.astype('int'))
  File "/home/tak/anaconda3/lib/python3.5/site-packages/numpy/lib/shape_base.py", line 91, in apply_along_axis
    res = func1d(arr[tuple(i.tolist())], *args, **kwargs)
  File "/home/tak/anaconda3/lib/python3.5/site-packages/sklearn/ensemble/voting_classifier.py", line 192, in <lambda>
    weights=self.weights)),
ValueError: The first argument of bincount must be non-negative
```

As the error suggests, only non-negative class labels can be used and hence breaks down when using it in the context of outlier detection where usually the outlier is identified by a -1.

The issue is resolved when after individual classifier predictions are done, the negative class labels are subsituted to some positive value and once the majority has been decided, reverse subsitution is done. Something like- 

```
def predict(self, X):
        check_is_fitted(self, 'estimators_')
        if self.voting == 'soft':
            maj = np.argmax(self.predict_proba(X), axis=1)

        else:  # 'hard' voting
            predictions = self._predict(X)

            # My addition
            predictions[predictions == -1] = 9

            maj = np.apply_along_axis(lambda x:
                                      np.argmax(np.bincount(x,
                                                weights=self.weights)),
                                      axis=1,
                                      arr=predictions.astype('int'))
        # My addition
        maj[maj == 9] = -1

        # Comment the following line
        # maj = self.le_.inverse_transform(maj)

        return maj
```

I had to comment the inverse_transform() function applied on the label encoders for this to work.
user: vishaltak
issue_8447: state: open
title: NMF with missing data
body: As far as I can tell, there is currently no way to do NMF (or other decomposition methods) with missing data. This could be useful for at least a couple of reasons:

1. Data imputation: Using factorization to fill in missing data can be a more refined technique than other data imputation schemes. It can also serve as the basis for recommender systems.

2. Cross-validation: Even when all data is present, holding out a random subset of elements in the matrix to be factored gives a way to evaluate model fit -- which can in turn be used to set hyperparameters (e.g. rank of decomposition, regularization strengths, etc).

Both of these use cases are explained (and implemented) quite nicely in R's `NNLM` package:
https://cran.r-project.org/web/packages/NNLM/vignettes/Fast-And-Versatile-NMF.html
(See sections "Missing value imputation and application in recommendation system" and "Determine rank k via missing value imputation").

At that API level, this might just look like assuming `np.nan` values are missing data.

Conceptually, it seems like it would be simple. In the NMF objection function, when computing `||X - WH||_F`, simply skip the missing values in the sum over elements in the Frobenius norm.

At the level of implementation in the solvers, I have no idea how hard this would be, as I'm not very well-acquainted with the code base.
user: jwittenbach
issue_8445: state: open
title: All kernel regression methods should accept precomputed Gram matrices
body: SVR has an option 
```
    kernel='precomputed'
```
If this is chosen, then the X array passed to the fit method is the Gram matrix of the training examples. This option should also be available for GPR and KRR.

Here is some motivation. Many machine learning projects begin by defining a feature set, and many algorithms intrinsically require a vector of real numbers for each sample. e.g. linear regression. 

Kernel methods take a different approach: the modeller supplies a function, the "kernel", which measures the similarity between two samples. This need not be expressed in terms of features. For example, it is easy to say what it means for two DNA sequences to be similar, but hard to reduce a DNA sequence to a vector of features. So SVR correctly is willing to build a model with:
```
model = svm.SVR(kernel='precomputed') 
model.fit( kernel(training.molecule),  training.y.values)
```
(My current project is a cheminformatic one.) 

It would be great if the other kernel methods supported this. At present, they require a kernel function which they pass the X values to, after checking that X is an array of floats. So some of the value of kernel methods is denied.
user: chrishmorris
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_8417: state: open
title: Give some multi-output classifier/regressor first order citizenship
body: Currently, all multi-output classifier/regressor are treated as second order citizenship.
every classifier/regressor are being trained separately using joblib. the data copy overhead is big. which is not necessary. we can follow multi-class classifier to define a vectorized loss function/grad/hess and hopefully should be faster than current implementation.

I've only tested with tensorflow one layer and currently multiouput. which tensorflow is way faster..

I was thinking to help on this, if you guys think it's necessary :) 
user: yupbank
issue_8409: state: open
title: Bug: BaggingClassifier for multiclass usage
body: Hi, 

The BaggingClassifier does not check if the number of classes of the random drawn samples for one of its estimators matches the number of classes in the dataset, resulting in an error message when its predict method is used: 

```
---------------------------------------------------------------------------
Sub-process traceback:
---------------------------------------------------------------------------
ValueError                                         Thu Feb 16 11:17:13 2017
PID: 25287                                    Python 2.7.5: /usr/bin/python
...........................................................................
.local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py in __call__(self=<sklearn.externals.joblib.parallel.BatchedCalls object>)
    126     def __init__(self, iterator_slice):
    127         self.items = list(iterator_slice)
    128         self._size = len(self.items)
    129
    130     def __call__(self):
--> 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
        func = <function _parallel_predict_proba>
        args = ([SVC(C=100, cache_size=200, class_weight=None, co...5695085, shrinking=True,
  tol=0.001, verbose=10), SVC(C=100, cache_size=200, class_weight=None, co...8919740, shrinking=True,
  tol=0.001, verbose=10)], [array([   0,    1,    2, ..., 2045, 2046, 2047]), array([   0,    1,    2, ..., 2045, 2046, 2047])], memmap([[  1.21488877e-01,  -6.84937861e-01,  -5...022708e-05,   1.09372859e-06,  -8.55540498e-06]]), 1000)
        kwargs = {}
        self.items = [(<function _parallel_predict_proba>, ([SVC(C=100, cache_size=200, class_weight=None, co...5695085, shrinking=True,
  tol=0.001, verbose=10), SVC(C=100, cache_size=200, class_weight=None, co...8919740, shrinking=True,
  tol=0.001, verbose=10)], [array([   0,    1,    2, ..., 2045, 2046, 2047]), array([   0,    1,    2, ..., 2045, 2046, 2047])], memmap([[  1.21488877e-01,  -6.84937861e-01,  -5...022708e-05,   1.09372859e-06,  -8.55540498e-06]]), 1000), {})]
    132
    133     def __len__(self):
    134         return self._size
    135

...........................................................................
.local/lib/python2.7/site-packages/sklearn/ensemble/bagging.py in _parallel_predict_proba(estimators=[SVC(C=100, cache_size=200, class_weight=None, co...5695085, shrinking=True,
  tol=0.001, verbose=10), SVC(C=100, cache_size=200, class_weight=None, co...8919740, shrinking=True,
  tol=0.001, verbose=10)], estimators_features=[array([   0,    1,    2, ..., 2045, 2046, 2047]), array([   0,    1,    2, ..., 2045, 2046, 2047])], X=memmap([[  1.21488877e-01,  -6.84937861e-01,  -5...022708e-05,   1.09372859e-06,  -8.55540498e-06]]), n_classes=1000)
    130     for estimator, features in zip(estimators, estimators_features):
    131         if hasattr(estimator, "predict_proba"):
    132             proba_estimator = estimator.predict_proba(X[:, features])
    133
    134             if n_classes == len(estimator.classes_):
--> 135                 proba += proba_estimator
        proba = array([[ 0.00130233,  0.00013968,  0.00144125, ....  0.00016293,
         0.00010567,  0.00053245]])
        proba_estimator = array([[  1.02577963e-03,   3.75469340e-04,   9....362413e-05,   1.45631109e-04,   3.04322015e-04]])
    136
    137             else:
    138                 proba[:, estimator.classes_] += \
    139                     proba_estimator[:, range(len(estimator.classes_))]

ValueError: operands could not be broadcast together with shapes (8009,1000) (8009,999) (8009,1000)
___________________________________________________________________________
```

It would be nice to get a warning message, if the number of classes used to train an estimator in the BaggingClassifier would not match the overall number of classes in the trainingsset. 



user: 4thedocs
issue_8381: state: open
title: Error in using multi-label classification in partial_fit() in OvR
body: - StackOverflow Question: https://stackoverflow.com/questions/42280439/multi-label-out-of-core-learning-for-text-data-valueerror-on-partial-fit

#### Description
When using OneVsRestClassifier() with partial_fit() method, errors are thrown. When using fit(), no errors are thrown and everything works.

#### Steps/Code to Reproduce
```
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.multiclass import OneVsRestClassifier
import numpy as np

categories = ['a','b','c']
X = ["This is a test", "This is another attempt", "And this is a test too!"]
Y = [['a', 'b'],['b', 'c'],['a', 'b']] 

mlb = MultiLabelBinarizer(classes=categories)
vectorizer = HashingVectorizer(decode_error='ignore', n_features=2 ** 18,         non_negative=True)
clf = OneVsRestClassifier(MultinomialNB(alpha=0.01))

X_train = vectorizer.fit_transform(X)
Y_train = mlb.fit_transform(Y)

- Case1
clf.partial_fit(X_train, Y_train, categories)
- Case2
clf.partial_fit(X_train, Y_train, mlb.transform(Y))
```
#### Description of code

- Case1   Using classes=categories without transforming 
```partial_fit(X_train, Y_train, classes=categories)```

        ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

- Case2   Using classes=mlb.transform(categories) i.e. after transforming from same multilabelbinarizer
```partial_fit(X_train, Y_train, classes=mlb.transform(categories))```

         ValueError: The object was not fitted with multilabel input.

#### Expected Results
No error is thrown as when using fit().

#### Actual Results
- Case1
> Traceback
 (most recent call last):
  File "/path_to_module/Check.py", line 18, in <module>
    clf.partial_fit(X_train, Y_train, categories)
  File "/library/python2.7/dist-packages/sklearn/multiclass.py", line 260, in partial_fit
    if np.setdiff1d(y, self.classes_):
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

- Case2
> Traceback
 (most recent call last):
  File "/path_to_module/Check.py", line 18, in <module>
    clf.partial_fit(X_train, Y_train, mlb.transform(Y))
  File "/library/python2.7/dist-packages/sklearn/multiclass.py", line 265, in partial_fit
    Y = self.label_binarizer_.transform(y)
  File "/library/python2.7/dist-packages/sklearn/preprocessing/label.py", line 329, in transform
    raise ValueError("The object was not fitted with multilabel"
ValueError: The object was not fitted with multilabel input.

#### Observation
- In Case1, the error is because https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/multiclass.py#L260 is returning an array of booleans whereas it expects a single boolean value hence the error. But couldnt find what to do about it. How to pass Y or classes into it.

- In Case2, the error occurs because partial_fit() calls the _check_partial_fit_first_call() function which sets the clf.classes_ in a different way using unique_labels() as seen here https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/multiclass.py#L308.
These unique labels are passed to clf.label_binarizer_ in this line https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/multiclass.py#L258, which leads to it assuming type of targets as 'multiclass' whereas actual type of target is multilabel, hence this error.
The fit() method handles classes_ in a different way (doesnt use unique_labels) and hence everything works correctly 

#### Versions
Linux-3.16.0-77-generic-x86_64-with-Ubuntu-14.04-trusty
('Python', '2.7.6 (default, Oct 26 2016, 20:30:19) \n[GCC 4.8.4]')
('NumPy', '1.12.0')
('SciPy', '0.18.1')
('Scikit-Learn', '0.18.1')


user: vivekk0903
issue_8370: state: open
title: API Freezing estimators
body: TODO: motivate freezing: pipeline components, calibration, transfer/semisupervised learning

This should probably be a SLEP, but I just want it saved somewhere.

Features required for estimator freezing:

* `clone` must have `is isinstance(obj, FrozenModel): return obj` (or do so via class polymorphism / `singledispatch`)
* `FrozenModel` delegates all attribute access (get, set, del) to its wrapped estimator (except where specified)
  * hence its estimator cannot be accessible at `FrozenModel().estimator` but at some more munged name.
* `FrozenModel` has `def fit(self, *args, **kwargs): return self`
* `FrozenModel` has `def fit_transform(self, *args, **kwargs): return fit(self, *args, **kwargs).transform(self, args[0])` (and similar for `fit_predict`?)
* `isinstance(freeze(obj), type(obj)) == True` and `isinstance(freeze(obj), FrozenModel) == True`
  * since this is determined from `type(freeze(obj))` (excluding `__instancecheck__`, which seems irrelevant), this appears to be the hardest criterion to fulfill
  * seems to entail use of a mixin, class created in closure, setting `__class__` (!), overloading of `__reduce__`, help! I think I've gone down the wrong path!!
* must behave nicely with `pickle` and `copy.[deep]copy`
* `freeze(some_list)` will freeze every element of the list
user: jnothman
issue_8359: state: open
title: Linear Kernel for Gaussian Processes
body: A simple linear kernel is lacking in Gaussian Processes, that is `K(x, x') = (x - c)(x' - c)

From http://www.cs.toronto.edu/~duvenaud/cookbook/index.html it is useful in modelling linear functions (obviously) and in combination with other functions such as periodic.


user: MechCoder
issue_8351: state: open
title: Initial attributes are modified in FeatureUnion
body: The [attributes](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/pipeline.py#L658) passed in `__init__` of FeatureUnion are modified during fitting, transgressing the scikit-learn API.

The API should be changed in accordance to discussion in #8157 

#### Related issues and PR

#8350 
#8157 

user: glemaitre
issue_8350: state: open
title: [MRG] FIX Modify the API of Pipeline and FeatureUnion to match common scikit-learn estimators conventions
body: #### Reference Issue

#8157 

#### What does this implement/fix? Explain your changes.

Modify the Pipeline API such that `steps` given by the user at initialization is not modified.

#### Any other comments?

* `steps` is replaced by `steps_` at fit time to follow scikit-learn API.
* `named_steps` has the same behaviour as before with a `FutureWarning`. It will return unfitted estimators from `steps`.
* `named_steps_` returns fitted estimators from `steps_` .
* `steps` becomes `_steps` in order to create property and setter `steps`.

Update: `FeatureUnion` has a nearly identical design issue.
user: glemaitre
labels: Needs Decision: Requires decision
issue_8349: state: open
title: Bug in bfgs gradient computation of MLPRegressor with multiple output neurons
body: When implementing a special Neural Network based on MLPRegressor, I found the following problem when using bfgs training and multiple output neurons (I did not look into the other training methods):
- The 'squared_loss' implementation uses np.mean to compute the overall loss. Thus, the method divides by the number of samples and the number of output neurons/features included in the dimensions of y_true - y_pred.
- The gradient computations do not include the number of output neurons. Gradients are only divided by the number of samples (_compute_loss_grad)
Overall, this leads to the fact, that the gradient has a wrong scaling by the number of output neurons.
As the search direction is still alright, this does not cause too much pain. Still, it should be fixed.

In case this is not clear, I can see that I create a minimal example.

Cheers!

#### Versions
>>> import platform; print(platform.platform())
Linux-3.16.0-4-amd64-x86_64-with-debian-8.5
>>> import sys; print("Python", sys.version)
('Python', '2.7.9 (default, Mar  1 2015, 12:57:24) \n[GCC 4.9.2]')
>>> import numpy; print("NumPy", numpy.__version__)
('NumPy', '1.10.4')
>>> import scipy; print("SciPy", scipy.__version__)
('SciPy', '0.14.0')
>>> import sklearn; print("Scikit-Learn", sklearn.__version__)
('Scikit-Learn', '0.18.1')


user: ghost
issue_8337: state: open
title: Add conditional inference trees
body: I think it would be a good idea to add conditional inference trees. From my understanding, they tend to provide better models when using a mixture of categorical and continuous values, or data with many duplicate values.

There was some discussion here but no follow up, I think;
https://sourceforge.net/p/scikit-learn/mailman/message/31141698/
user: amueller
issue_8317: state: open
title: [WIP] ENH arbitrary diagnostic_func callback in GridSarchCV
body: This is a proof of concept for having an arbitrary callback function in `GridSearchCV` (and other CVs in the future) to return non-standard metrics, classification reports, coefficients, etc.; or to dump the model to disk, for instance. This gets around some of the limitations, and the complex engineering, of multiple metric support as in #7388; see https://github.com/scikit-learn/scikit-learn/issues/1837#issuecomment-277912593

Note that the modifications to the example here are currently rubbish just to be played with, not a true proposed change to that example.
user: jnothman
issue_8303: state: open
title: Prediction interval for MLPRegressor
body: Hi there,

Is there any plan to add prediction interval to a MLPRegressor? I'm using a MLPRegressor to model system response, and I need to a 95% prediction interval, if system deviates from this range, an alarm will be thrown. 

Thanks!
user: wqp89324
issue_8243: state: open
title: Using Several Parameters with GridSearchCV
body: When using (for *example*) the following transformators

 -  ``CountVectorizer``
 - ``TruncatedSVD``
 - ``SelectKBest``

with ``GridSearchCV`` it happens that it chooses a number for ``n_features`` for ``CountVectorizer`` that is less than ``n_components`` for ``TruncatedSVD``  or ``k`` for ``SelectKBest``.

This leads to an **error**: ``ValueError: n_components must be < n_features``

For ``SelectKBest`` I found a temporary solution:

    class SelectAtMostKBest(SelectKBest):
        def _check_params(self, X, y):
            if not (self.k == "all" or 0 <= self.k <= X.shape[1]):
                self.k = "all"

But there is no equivalent for ``TruncatedSVD``.
Is this behaviour intended? If yes, what can I do about this?
user: ScientiaEtVeritas
issue_8238: state: open
title: retain original indexes
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description

I am extremely surprised to find that sklearn models reset the original indexes of dataframes that are input for fitting/transforming. I think in the future all models should have the ability to retain the original indexes as they are handed to the models. This would help a lot while deploying these models in real world settings. It is very clumsy to have to save out the index and then join it back in at the end, hoping that the models did not change the order of the rows along the way. My suggestion is that every model have an option to "retain_index=True/False".
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->

#### Steps/Code to Reproduce
<!--
Example:
```
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->


<!-- Thanks for contributing! -->

user: jeweinberg
issue_8230: state: open
title: [MRG] GridSearchCV.use_warm_start parameter for efficiency
body: Alternative to #8226, provides a generic CV optimisation making use of `warm_start`.

The example, modified from #8226, shows the benefit of using this for optimising GBRT `n_estimators`.

![times](https://cloud.githubusercontent.com/assets/78827/22247073/0cc06fe4-e28c-11e6-9369-0547d083e969.png)

* [x] Basic implementation version 2 (`use_warm_start` takes a string/list) as at 1-Feb-2017
* [x] Parameter docstring
* [x] Tests
* [x] Example
* [x] Narrative docs
user: jnothman
labels: Needs Decision: Requires decision
issue_8223: state: open
title: Perform CV and get results from scoring and prediction
body: Hi all together,

I am wondering how you are doing this. If I have a large dataset an want to do CV I have to fit a model twice if I want to get the prediction and the score results. Is it also possible to get both at a time? If not, I would vote to make this possible as it saves lot's of time. I would also like to contribute if there is work to be done on this.

Best Regards
Manuel
user: manuSrep
issue_8212: state: open
title: different coefficients and intercepts obtained with different solvers of LinearDiscriminantAnalysis
body: #### Description
different coefficients and intercepts obtained with different solvers of `LinearDiscriminantAnalysis`. Seems that `svd` and `lsqr` give same results, compared to `eigen`. Also I noticed that in testing code of this <https://github.com/scikit-learn/scikit-learn/blob/0.18.1/sklearn/tests/test_discriminant_analysis.py#L141-L143>, the number of decimals compared is only 1, which is probably why the test could pass.

I also tried multiple classes, and the result of three solvers are all different.

I only tried sklearn 0.18, but it should be the same for 0.18.1.

#### Steps/Code to Reproduce

```python
import sklearn
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
import numpy as np

rng_state = np.random.RandomState(seed=0)
X = rng_state.randn(1000, 5)*np.array([1,11,20,4,20])
y = rng_state.randint(0, 2, size=(1000,))
clf_1 = LinearDiscriminantAnalysis(solver="svd")
clf_1 = clf_1.fit(X, y)
clf_2 = LinearDiscriminantAnalysis(solver="eigen")
clf_2 = clf_2.fit(X, y)
clf_3 = LinearDiscriminantAnalysis(solver="lsqr")
clf_3 = clf_3.fit(X, y)

print(clf_1.coef_-clf_2.coef_,clf_1.coef_-clf_3.coef_)
print(clf_1.intercept_-clf_2.intercept_,clf_1.intercept_-clf_3.intercept_)
```

#### Expected Results
last few numbers should be all close to zero.

#### Actual Results
```
(array([[ 0.30953158, -0.05400618,  0.01572382,  0.13094748, -0.02656968]]), array([[  7.67230451e-05,  -1.33864168e-05,   3.89743481e-06,
          3.24577203e-05,  -6.58578056e-06]]))
(array([ 0.05393478]), array([  1.33687182e-05]))
```

#### Versions

```
Linux-2.6.32-279.14.1.el6.x86_64-x86_64-with-centos-6.2-Final
('Python', '2.7.12 | packaged by conda-forge | (default, Sep  8 2016, 14:22:31) \n[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]')
('NumPy', '1.11.2')
('SciPy', '0.18.1')
('Scikit-Learn', '0.18')
```

user: zym1010
issue_8206: state: open
title: [WIP] Add prediction strength method to determine number of clusters
body: ### Description
Prediction strength is a metric that measures the stability of a clustering algorithm and can be used to determine an optimal number of clusters without knowing the true cluster assignments. It has been introduced by [Tibshirani and Guenther (2005)](http://doi.org/10.1198/106186005X59243).

First, one splits the data into two parts (A) and (B). One obtains two cluster assignments, the first one using the centroids derived from the subset (A), and the second one using the centroids from the subset (B). Prediction strength measures the proportion of observation pairs that are assigned to the same clusters  according to both clusterings. The overall prediction strength is the minimum of this quantity over all predicted clusters.

The definition of prediction strength makes it difficult to integrate it into the existing GridSearchCV and metrics API, because a clusterer needs to trained twice, each time on a different subset of the data, and evaluated on a single subset.

### Implementation
Given two lists of cluster assignments, the function `prediction_strength_score` computes the actual score.
The class `PredictionStrengthGridSearchCV`extends `GridSearchCV` and modifies the way `_fit_and_score` operates. Instead of fitting an estimator on a training set and evaluating it on a test set (`_default_fit_and_score` function),  we need to fit and predict twice and compute the prediction score based on the two predictions (`_prediction_strength_fit_and_score` function). In addition, `PredictionStrengthGridSearchCV` uses the cross-validation generator twice, where the roles of training and test set are reversed the second time around. The final prediction strength for a given *k* and fold are the averages from both runs.

Moreover, `PredictionStrengthGridSearchCV` overrides how the best estimator is selected. The optimal number of clusters *k* is the largest *k* such that the corresponding prediction strength is above some threshold. Tibshirani and Guenther suggest a threshold in the range 0.8 to 0.9 for well separated clusters. If no configuration exceeds the threshold, a warning is displayed and the configuration with the highest prediction strength is selected.

`PredictionStrengthGridSearchCV` assumes that the `param_grid` parameter contains a key "n_clusters". 

### Open questions
1. The user could define additional parameters beside "n_clusters" in the parameter grid. For a given number of clusters, one would first need to choose the configuration that maximizes the prediction strength, and then choose the maximum *k* such that the maximum prediction strength is above a threshold.

2. Alternatively, one could remove "param_grid" and only offer the option to define a list of parameters for "n_clusters".

3. Finally, I had to add a flag `use_prediction_strength` to `_fit_and_score` to allow deviating from the common case: fit once and predict once. I welcome any suggestions on how this could be improved without duplicating too much code.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: sebp
labels: Needs Decision: Requires decision
issue_8195: state: open
title: Significance test for Adjusted Rand Score
body: Is there appetite for addition of a small function that performs a significance test for the adjusted Rand score given two label partitions (ie `labels_true` and `labels_pred`)?

[1] outlines  a Monte Carlo approximation approach for the significance testing of the adjusted Rand index. Essentially, this involves randomly sampling possible clustering assignments, given the same number of clusters and same number of samples in each cluster as compared to the input labels. 

I've written the code to do this, but not sure if it falls within the scope of Scikit Learn. I wanted to check here before doing the full documentation write-up and associated pull request.

> [1] Qannari EM, Courcoux P, Faye P. Significance test of the adjusted Rand index. Application to the free sorting task. Food Qual Prefer. 2014/3;32, Part A: 93–97.
user: andrewguy
issue_8194: state: open
title: Should all test cases set the random_state?
body: In [this](https://github.com/scikit-learn/scikit-learn/pull/7591) PR some newly added test cases made some unrelated tests fail. The culprit was mutating state due to missing `random_state`(s) in the unrelated tests' function/class invocations. @amueller also mention that:

> Ideally we would explicitly pass random states everywhere in all tests. To fix the test failure, maybe fix the random states in the test that's failing. You can also add it in the test that you added.

I made a simple python script [here](https://gist.github.com/nuffe/3afcc7d6682b05d71e59a99afcf0cbfe) that goes through each of `sklearn`'s test files and look for estimators/classes that take a `random_state` parameter, and see if this was passed in the invocation(s). If not then print out the class/function name followed by a list of line numbers with missing invocations.

On the latest master branch (*ca687ba*) there are 1 155 cases in 154 test files where a class/function that takes a `random_state` is called without setting it to the canonical `42`:

```bash
erik  scikit-learn$ find * -type f | grep "/test_.*py$" | python rcheck.py
sklearn/tree/tests/test_tree.py
	DecisionTreeClassifier [398, 1173, 1210, 1231, 1237, 1531]
	DecisionTreeRegressor [1532]
sklearn/tree/tests/test_export.py
        DecisionTreeClassifier [197, 212]
.
.
.
sklearn/model_selection/tests/test_search.py
	DecisionTreeClassifier [868]
	DecisionTreeRegressor [868]
	GroupShuffleSplit [242]
	KFold [241, 250, 491, 503, 919, 1172, 1193]
	LinearSVC [292, 356, 363, 376, 383]
	LinearSVCNoScore [101, 210]
	ParameterSampler [1093, 1096, 1113]
	RandomizedSearchCV [265, 704, 708, 753, 798, 839, 886, 946, 983]
	SGDClassifier [1126, 1141]
	SVC [292, 301, 305, 325, 333, 340, 348, 356, 363, 376, 383, 417, 441, 644, 647, 704, 708, 752, 753, 795, 798, 838, 839, 1003]
	StratifiedKFold [250, 919]
	StratifiedShuffleSplit [250]

Found 1155 missing in 154 files
```
The full output is [here](https://gist.github.com/nuffe/62fa22280117e0f90e988ee15aa2c966)

If desirable, I could work on a PR for a find/replace on all these cases? If so, then the scikit-learn contributing docs should perhaps be explicit in stating that all added test cases that invoke an estimator that takes a `random_state`, should also set it / or add some kind of flake plugin that checks this?




user: erikcs
labels: Needs Decision: Requires decision
issue_8157: state: open
title: RFC: Cloning estimators in pipeline
body: 
This is an RFC on the lack of cloning of estimators in the pipeline.

# Current situation

The `Pipeline` modifies in the fit method the estimators that are given in it's `steps` argument, in violation of the scikit-learn convention (bad bad coder).

# Specific issue raised

To implement caching of the fit of transformers reliably, cloning of them is crucial (https://github.com/scikit-learn/scikit-learn/pull/7990), this way the caching is dependent only on the model parameters of the transformers.

In the PR (#7990), the first attempt was to implement a new class, `CachedPipeline` that would behave like the `Pipeline` but clone the transformers and cache them. The drawback is the multiplication of classes that are very similar, which makes features harder to discover and give surprises as there is a subttle difference between `Pipeline` and `CachedPipeline`.

# Proposal

The proposal (put forward IRL by @ogrisel, but that has my favor too) is to deprecate the fact that `pipeline.steps` is modified and introduce a `.steps_` attribute (and a `.named_steps_` attribute). That way we could merge `Pipeline` and `CachedPipeline`.

# Implementation

The difficulty is the deprecation path, as often. Two options:

1. Make `steps` and `named_steps` be properties, and add a warning upon access. Make it so that, for two releases, they return the modified estimators, ie what is stored in `steps_` and `named_steps_`. In two releases, `named_steps` dies in favor of `named_steps_`, and we remove the properties.

2. Create a new class, for instance `Pipe`, that has the new logic with optional caching.

I have no strong opinion on which deprecation path is best. Option 1 is more work but may less intrusive on the users.


user: GaelVaroquaux
issue_8105: state: open
title: [WIP] Progress Logger
body: #### Reference Issue
This is start of work to address the issues in #78 #7574

#### What does this implement/fix? Explain your changes.
This adds an initial draft of the ProgIter object. This object 
wraps around long-running loops and reports progress in a simple but customization way. 
The purpose is meant to simplify writing progress messages and create a construct that can simply be dropped into existing code.

I've added in usage of this object into kmeans++ and MiniBatchKmeans to address my original use case. The case with MiniBatchKmeans requires a bit more customization because other useful messages in the loop. However, the case with kmeans++ where the change is extremely minimal and really demonstrates how this can just be dropped into existing code. 


#### Any other comments?

While this feature does work end-to-end it is not complete. 
Because this is a work in progress I have left some debugging code, references to my utility library (utool), and documentation constructs that I use in my coding workflow to quickly test things. 

I don't see myself able to work much on this feature in the near future, but I did want to push what I had so far because (a) just to put it out there, (b) so I don't lose it, and (c) in case it is useful to someone else. 


* **Basic Usage**
Basic usage of the object looks like this: 

```python
    >>> from sklearn.externals.progiter import ProgIter
    >>> def is_prime(n):
    ...     return n >= 2 and not any(n % i == 0 for i in range(2, n))
    >>> for n in ProgIter(range(10000)):
    >>>     # do some work
    >>>     is_prime(n)
    10000/10000... rate=13294.94 Hz, eta=0:00:00, total=0:00:00, wall=13:34 EST 
```
By default the output is continually updated by clearing the previous line, so at the end only the last progress message shows.

* **Demo Function**
The progiter module itself contains a small demo function to demonstrate how it works. Running the demo results in this output: 

```
-----
Demo #0: progress can be disabled and incur essentially 0 overhead
However, the overhead of enabled progress is minimal and typically insignificant
this is verbosity mode verbose=0

tic(u'demo0')
...toc(u'demo0')=0.6284s

-----
Demo #1: progress is shown by default in the same line
this is verbosity mode verbose=1

tic(u'demo1')
demo1 1000/1000... rate=1584.12 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
...toc(u'demo1')=0.6314s

-----
Demo #2: clearline=False prints multiple lines.
Progress is only printed as needed
Notice the adjustment behavior of the print frequency
this is verbosity mode verbose=2

tic(u'demo2')
demo2    0/1000... rate=0.00 Hz, eta=?, total=0:00:00, wall=13:20 EST 
demo2    1/1000... rate=2702.52 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
demo2  257/1000... rate=1522.38 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
demo2  642/1000... rate=1560.16 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
demo2 1000/1000... rate=1573.36 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
...toc(u'demo2')=0.6357s

-----
Demo #3: Adjustments can be turned off to give constant feedback
this is verbosity mode verbose=3

tic(u'demo3')
demo3    0/1000... rate=0.00 Hz, eta=?, total=0:00:00, wall=13:20 EST 
demo3  100/1000... rate=1476.12 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
demo3  200/1000... rate=1427.07 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
demo3  300/1000... rate=1479.27 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
demo3  400/1000... rate=1505.12 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
demo3  500/1000... rate=1491.30 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
demo3  600/1000... rate=1515.48 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
demo3  700/1000... rate=1512.45 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
demo3  800/1000... rate=1514.54 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
demo3  900/1000... rate=1515.45 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
demo3 1000/1000... rate=1516.72 Hz, eta=0:00:00, total=0:00:00, wall=13:20 EST 
...toc(u'demo3')=0.6594s
```

* **Usage in MiniBatchKMeans**
An example showing how this works in the context of MiniBatchKMeans is here: 

```python
        >>> from sklearn.cluster import MiniBatchKMeans
        >>> from sklearn.datasets.samples_generator import make_blobs
        >>> import numpy as np
        >>> n_clusters = 4000
        >>> X, true_labels = make_blobs(n_samples=int(1E5), centers=n_clusters,
        ...                             cluster_std=1., random_state=42)
        >>> mbkm = MiniBatchKMeans(n_clusters=n_clusters,
        ...                        init_size=3 * n_clusters, n_init=2,
        ...                        random_state=0, verbose=2).fit(X)
        >>> print('mbkm.labels_ = %r' % (mbkm.labels_,))
```

First with verbose=2, which means the ProgIter object will not clear progress lines and always just append the next line to stdout.

```
Init cluster centers with k-means++
kmeans++    0/4000... rate=0.00 Hz, eta=?, total=0:00:00, wall=13:24 EST 
kmeans++    1/4000... rate=192.34 Hz, eta=0:00:20, total=0:00:00, wall=13:24 EST 
kmeans++  192/4000... rate=931.83 Hz, eta=0:00:04, total=0:00:00, wall=13:24 EST 
kmeans++  480/4000... rate=1034.86 Hz, eta=0:00:03, total=0:00:00, wall=13:24 EST 
kmeans++ 1117/4000... rate=1084.50 Hz, eta=0:00:02, total=0:00:01, wall=13:24 EST 
kmeans++ 1125/4000... rate=1084.81 Hz, eta=0:00:02, total=0:00:01, wall=13:24 EST 
kmeans++ 2250/4000... rate=1107.09 Hz, eta=0:00:01, total=0:00:02, wall=13:24 EST 
kmeans++ 2260/4000... rate=1107.16 Hz, eta=0:00:01, total=0:00:02, wall=13:24 EST 
kmeans++ 3390/4000... rate=1112.93 Hz, eta=0:00:00, total=0:00:03, wall=13:24 EST 
kmeans++ 4000/4000... rate=1114.58 Hz, eta=0:00:00, total=0:00:03, wall=13:24 EST 
Inertia for init 1/2: 169.317605
Init cluster centers with k-means++
kmeans++    0/4000... rate=0.00 Hz, eta=?, total=0:00:00, wall=13:24 EST 
kmeans++    1/4000... rate=3412.78 Hz, eta=0:00:01, total=0:00:00, wall=13:24 EST 
kmeans++  257/4000... rate=975.97 Hz, eta=0:00:03, total=0:00:00, wall=13:24 EST 
kmeans++  642/4000... rate=1059.49 Hz, eta=0:00:03, total=0:00:00, wall=13:24 EST 
kmeans++ 1123/4000... rate=1044.57 Hz, eta=0:00:02, total=0:00:01, wall=13:24 EST 
kmeans++ 2050/4000... rate=1060.59 Hz, eta=0:00:01, total=0:00:01, wall=13:24 EST 
kmeans++ 2160/4000... rate=1063.67 Hz, eta=0:00:01, total=0:00:02, wall=13:24 EST 
kmeans++ 3240/4000... rate=1074.09 Hz, eta=0:00:00, total=0:00:03, wall=13:24 EST 
kmeans++ 3285/4000... rate=1074.54 Hz, eta=0:00:00, total=0:00:03, wall=13:24 EST 
kmeans++ 4000/4000... rate=1082.62 Hz, eta=0:00:00, total=0:00:03, wall=13:24 EST 
Inertia for init 2/2: 169.461913
Begining mini-batch iterations
minibatch      0/100000... rate=0.00 Hz, eta=?, total=0:00:00, wall=13:24 EST 
minibatch      1/100000... rate=24.79 Hz, eta=1:07:13, total=0:00:00, wall=13:24 EST inertias: batch=0.024289, ewa=0.024289 
[MiniBatchKMeans] Reassigning 50 cluster centers.
minibatch     12/100000... rate=37.27 Hz, eta=0:00:00, total=0:00:00, wall=13:24 EST inertias: batch=0.022055, ewa=0.024292 
Converged (lack of improvement in inertia) at iteration 12/100000
Computing label assignment and total inertia
labels inertia    0/1000... rate=0.00 Hz, eta=?, total=0:00:00, wall=13:24 EST 
labels inertia    1/1000... rate=348.57 Hz, eta=0:00:02, total=0:00:00, wall=13:24 EST 
labels inertia  257/1000... rate=382.73 Hz, eta=0:00:01, total=0:00:00, wall=13:24 EST 
labels inertia  382/1000... rate=382.87 Hz, eta=0:00:01, total=0:00:00, wall=13:24 EST 
labels inertia  764/1000... rate=389.35 Hz, eta=0:00:00, total=0:00:01, wall=13:24 EST 
labels inertia  792/1000... rate=389.13 Hz, eta=0:00:00, total=0:00:02, wall=13:24 EST 
labels inertia 1000/1000... rate=390.99 Hz, eta=0:00:00, total=0:00:02, wall=13:24 EST 
mbkm.labels_ = array([ 972, 1783,  797, ..., 3203, 3363, 2020], dtype=int32)

```

Verbosity 2 can be a bit much, so when verbosity=1 a clearline sequence will tell the terminal to delete the previous verbosity line before it prints the next one. This has a nice where you see a single line in the terminal update every once in awhile. Obviously I can't show updates in static text, but at the end of the script it looks like this: 

```
Init cluster centers with k-means++
kmeans++ 4000/4000... rate=1260.10 Hz, eta=0:00:00, total=0:00:03, wall=13:28 EST 
Inertia for init 1/2: 169.317605
Init cluster centers with k-means++
kmeans++ 4000/4000... rate=1243.40 Hz, eta=0:00:00, total=0:00:03, wall=13:28 EST 
Inertia for init 2/2: 169.461913
Begining mini-batch iterations
minibatch      1/100000... rate=24.85 Hz, eta=1:07:03, total=0:00:00, wall=13:28 EST inertias: batch=0.024289, ewa=0.024289 
[MiniBatchKMeans] Reassigning 50 cluster centers.
minibatch     12/100000... rate=36.89 Hz, eta=0:00:00, total=0:00:00, wall=13:28 EST inertias: batch=0.022055, ewa=0.024292 
Converged (lack of improvement in inertia) at iteration 12/100000
Computing label assignment and total inertia
labels inertia 1000/1000... rate=451.42 Hz, eta=0:00:00, total=0:00:02, wall=13:28 EST 
mbkm.labels_ = array([ 972, 1783,  797, ..., 3203, 3363, 2020], dtype=int32)
```


user: Erotemic
labels: Needs Decision: Requires decision
issue_8100: state: open
title: Helpers for managing different set of classes
body: It seems like half the bugs we've solved in the past couple of months surround problems of having different sets of classes, in the context of:
* cross-validation splitting that yields different subsets of classes in different training or testing subsets (and hence issues in alignment of class-wise outputs from `predict_proba`, `decision_function` or metrics, or in normalising macro-averaged scores)
* `partial_fit` where `classes` are specified upfront, but then repeated calls need matching to those classes
* `warm_start` where `classes_` from the first fit must be identical to the set of classes in `y` in each call to `fit()`

These are all subtly different problems, but at the moment it seems like we're handling them on an ad-hoc (and too often a post-hoc) basis.

It would be amazing if someone could review these issues and identify where either API changes (`classes` as a constructor parameter to classifiers has been suggested) or helper utilities might help avoid these issues in the future.
user: jnothman
issue_8098: state: open
title: Bug in metrics.classification._check_targets?
body: #### Description
I came across this issue while checking a test in `test_matthews_corrcoef`. In this test two label vectors are defined. The groundtruth vector is filled with `a`'s and `b`'s. The prediction vector is filled with `-1`'s and `0`'s. The `_check_targets` in `sklearn.metrics.classification` reports these vectors are binary when it seems to me it should be multiclass instead.

If this is the correct behavior and these pair of vectors should be classified as binary, then there would need to be some association between the numbers in the prediction vector and letters in the groundtruth vector and this association should be captured in the `LabelEncoder` used latter in the `matthews_corrcoef` function, but it is not. So, there is either a bug in `_check_inputs` or the `LabelEncoder`. 


#### Steps/Code to Reproduce

The following code takes the bit of the `test_matthews_corrcoef` and `matthews_corrcoef` functions that produce the odd states. 

```
    from sklearn.preprocessing import label_binarize, LabelEncoder
    from sklearn.metrics.classification import _check_targets
    import numpy as np
    rng = np.random.RandomState(0)
    y_true = ["a" if i == 0 else "b" for i in rng.randint(0, 2, size=20)]
    y_pred = label_binarize(y_true, ["a", "b"]) * -1
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)
    lb = LabelEncoder()
    lb.fit(np.hstack([y_true, y_pred]))
    y_true = lb.transform(y_true)
    y_pred = lb.transform(y_pred)
    print('y_type = %r' % (y_type,))
    print('lb.classes_ = %r' % (lb.classes_,))
    print('y_true = %r' % (y_true,))
    print('y_pred = %r' % (y_pred,))
```

The initial values of y_true and y_pred are: 
```
y_true = [u'a', u'b', u'b', u'a', u'b', u'b', u'b', u'b', u'b', u'b', u'b', u'a', u'a', u'b', u'a', u'a', u'a', u'a', u'a', u'b']
y_pred = [[0], [-1], [-1], [0], [-1], [-1], [-1], [-1], [-1], [-1], [-1], [0], [0], [-1], [0], [0], [0], [0], [0], [-1]]
```

After they leave _check_inputs they are
```
y_true = array([u'a', u'b', u'b', u'a', u'b', u'b', u'b', u'b', u'b', u'b', u'b',
       u'a', u'a', u'b', u'a', u'a', u'a', u'a', u'a', u'b'], 
      dtype='<U1')

y_pred =  array([ 0, -1, -1,  0, -1, -1, -1, -1, -1, -1, -1,  0,  0, -1,  0,  0,  0, 0,  0, -1])
```

At the end the values are: 

#### Expected Results
```
y_type = 'multiclass'
lb.classes_ = array([u'-1', u'0', u'a', u'b'], 
      dtype='<U21')
y_true = array([2, 3, 3, 2, 3, 3, 3, 3, 3, 3, 3, 2, 2, 3, 2, 2, 2, 2, 2, 3])
y_pred = array([1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0])
```

OR 

```
y_type = 'binary'
lb.classes_ = array([u'a', u'b'], dtype='<U21')
y_true = array([0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1])
y_pred = array([1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0])
```

#### Actual Results

```
y_type = 'binary'
lb.classes_ = array([u'-1', u'0', u'a', u'b'], 
      dtype='<U21')
y_true = array([2, 3, 3, 2, 3, 3, 3, 3, 3, 3, 3, 2, 2, 3, 2, 2, 2, 2, 2, 3])
y_pred = array([1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0])
```

It doesn't make sense to me that `_check_inputs` would output that these cases are binary when the LabelEncoder ends up encoding 4 classes and transforming them as such. 

Fortunately, this behavior does not cause `matthews_corrcoef` to produce the wrong value because the vectors are centered in the next line of code which effectively undoes the bug. 

However, this is causing me issues because I'm attempting to extend `matthews_corrcoef` to the multiclass case (in PR #8094) and I need to be able to distinguish this case from the case where a predicted class exists that is not in the groundtruth or a groundtruth class exists that was not predicted. 

#### Thoughts on a fix

My idea for resolving this issue is to have `_check_inputs` check the type of a concatenated version of `y_true` and `y_pred`, however I'm not sure how to go about this with more complex types like `multilabel-indicator`. 

#### Versions
Linux-3.13.0-105-generic-x86_64-with-Ubuntu-14.04-trusty
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2]
NumPy 1.11.2
SciPy 0.18.0
Scikit-Learn 0.19.dev0
user: Erotemic
issue_8073: state: open
title: something strange in AffinityPropagation
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->  
Today I run this code about AffinityPropagation,but something strange happens. The data I used is generated by sklean.datasets. I don't know why the result in picture 7 and picture 8  is abnormal while the result in picture 9 is normal.



#### Steps/Code to Reproduce
<!--
Example:
```
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->
```
# -*- coding:utf-8 -*-

import numpy as np
import matplotlib.pyplot as plt
import sklearn.datasets as ds
import matplotlib.colors
from sklearn.cluster import AffinityPropagation
from sklearn.metrics import euclidean_distances

if __name__ == "__main__":

    N = 400
    centers = [[1, 2], [-1, -1], [1, -1], [-1, 1]]
    data, y = ds.make_blobs(N, n_features=2, centers=centers, cluster_std=[0.5, 0.25, 0.7, 0.5], random_state=0)

    m = euclidean_distances(data, squared=True)
    preference = -np.median(m)

    matplotlib.rcParams['font.sans-serif'] = [u'SimHei']
    matplotlib.rcParams['axes.unicode_minus'] = False

    plt.figure(figsize=(12, 9), facecolor='w')
    for i, mul in enumerate(np.linspace(1, 4, 9)):
        print mul
        p = mul * preference
        model = AffinityPropagation(affinity='euclidean', preference=p)
        af = model.fit(data)
        center_indices = af.cluster_centers_indices_
        n_clusters = len(center_indices)
        y_hat = af.labels_

        plt.subplot(3, 3, i+1)
        plt.title(u'Preference, %.2f, num_clusters:%d' % (p, n_clusters))
        clrs = []
        for c in np.linspace(16711680, 255, n_clusters):
            clrs.append('#%06x' % c)

        for k, clr in enumerate(clrs):
            cur = (y_hat == k)
            plt.scatter(data[cur, 0], data[cur, 1], c=clr, edgecolors='none')
            center = data[center_indices[k]]
            for x in data[cur]:
                plt.plot([x[0], center[0]], [x[1], center[1]], color=clr)
        plt.scatter(data[center_indices, 0], data[center_indices, 1], s=100, c=clrs, marker='*', edgecolors='k', zorder=2)
    plt.grid(True)
    plt.tight_layout()
    plt.suptitle(u'AffinityPropagation', fontsize=20)
    plt.subplots_adjust(top=0.92)
    plt.show()
```

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->
Actually I do not ensure this is a bug,but according to my experience the number of clusters in picture 7 and picture 8 should not be so large.

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
![result][1]


#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->
Windows-10-10.0.14393
('Python', '2.7.12 |Continuum Analytics, Inc.| (default, Jun 29 2016, 11:07:13) [MSC v.1500 64 bit (AMD64)]')
('NumPy', '1.11.2')
('SciPy', '0.18.1')
('Scikit-Learn', '0.18.1')

<!-- Thanks for contributing! -->

  [1]: https://raw.githubusercontent.com/wj1066/test/master/111.png
user: weijianzz
issue_8057: state: open
title: Nearest neighbors classifiers should support sparse multilabel Y
body: This should be a fairly straightforward translation of each output's mode to sparse CSC, then hstacking.
user: jnothman
issue_8055: state: open
title: Feature Importance by Class Attribute in GBRT
body: Are you open to expanding the functionality of GBRT to return feature importances for each class in a multinomial model, rather than just the sum importances over the entire model? E.g.,

```
def multi_class_importance(trained_model,output=0):
    
    total_importance = np.zeros((trained_model.n_features, ), dtype=np.float64)
    class_importances = {}

    for class_idx,class_label in enumerate(trained_model.classes_):

        class_ensemble = trained_model.estimators_[:, class_idx]

        class_importance = sum(tree.feature_importances_ 
                               for tree in class_ensemble) / len(class_ensemble)

        class_importances[class_label] = class_importance

        # Below returns comparable results to clf.feature_importances_
        # Except that it rounds to the 18th digit
        total_importance += class_importance / len(trained_model.classes_)
        
    if output == 0:
        return class_importances
    else:
        return total_importance
```
user: thommiano
issue_8049: state: open
title: OneVsOneClassifier decision function shape non-standard
body: For binary tasks, OvO has a shape of ``(n_samples, 2)`` which pretty much violates our standards.
I'm not sure what the best solution is apart from just breaking it as a bug-fix.
We could add a parameter and deprecate it and then remove the parameter if we really want.

The OVR classifier also has a potential issue where the decision_function for binary is ``(n_samples, 1)`` instead of ``(n_samples,)``. That also seems non-standard. I'm not sure we have other multi-output classifiers with a decision function, so I'm not entirely certain what the standard should be. All the multi-label ones do ``(n_samples,)`` according to the tests.

Found via #8022.
user: amueller
issue_8030: state: open
title: [WIP] Add SmartSplitter for handling categorical features in decision tree
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->

Add SmartSplitter: use local response proportion for categorical feature during decision tree node split process.

TODO:
- Add tests
- Run tests

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->
user: lilianweng
issue_8025: state: open
title: Add intersection/Jensen-Shannon kernel samplers
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
Additive chi2 kernel sampler seems to be not enough to handle many descriptors widely used in computer vision (e.g. RLH and Docstrum). Other kernels worth supporting are intersection and Jensen-Shannon kernels.
Adding support for those kernels is easy: samplers for them differ from AdditiveChi2Sampler only by preset sampling intervals and spectrum function.

user: ValeryAnisimovsky
issue_8023: state: open
title: Should cloning create equal parameters
body: I'm trying to make our common tests work on instances instead of classes in #8022 - this is a prerequisite to implement instance-level estimator tags and get rid of the hard-coded classes.
I ran into an *interesting* issue.

Instead of instantiating an estimator, I clone it. I expected that parameters after cloning are equal (``==``) to before cloning. They are not.

```python
from sklearn.cluster import AgglomerativeClustering
from sklearn.base import clone

agg = AgglomerativeClustering()
agg2 = clone(agg)

agg.memory == agg2.memory
```
> False

Question: should this condition be true? I think it's not actually an issue for changing the tests, but it seemed somewhat weird to me.

ping @GaelVaroquaux @jnothman @agramfort 

user: amueller
issue_8021: state: open
title: [MRG+1] Intersection/Jensen-Shannon kernel samplers
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
<!-- Example: Fixes #1234 -->
#8025

#### What does this implement/fix? Explain your changes.
This PR adds samplers for intersection and Jensen-Shannon kernels (in addition to additive chi2 kernel sampler).

#### Any other comments?
Most part of the AdditiveChi2Sampler class code has been moved to a new class BaseAdditiveHomogenousKernelSampler which is made a base class for AdditiveChi2Sampler, IntersectionSampler and JensenShannonSampler. Those three classes only override __init__ method which sets class-specific attributes.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: ValeryAnisimovsky
labels: Needs Decision: Requires decision
issue_8020: state: open
title: Randomness in LatentDirichletAllocation(learning_method="batch")
body: It looks to me as if there's randomness in the batch variant of the LDA in ``_em_step`` where ``random_init`` is set to ``True``. That seems incorrect to me.

In general I find the logic of the algorithm a bit hard to follow, I'm wondering whether we should either refactor it or implement batch and online in two entirely separate code paths.
user: amueller
issue_8012: state: open
title: cache_size > 2000 not recognized in SVM classifiers
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
Noticed this while using libSVM in sklearn - training an SVM when cache_size > 2000 or so on large problems does not seem to lead to any benefit/speed up. Looking at RAM usage, it shows that usage is still about 200MB (which is roughly the original dataset size, rather than the Kernel matrix size). Looks like the issue is in svm.cpp, where the cache size is set to(long int) cache_size*(1<<20). I suspect this overflows for cases where cache_size>2000.


Also, it might be a good idea to update the documentation to note that increasing `cache_size` does not guarantee faster training. 

#### Steps/Code to Reproduce
This example shows the issue with RAM usage (for `cache_size=4000`, the RAM usage is 77MB for this Python instance. However, RAM usage is 2079MB when running with `cache_size=4000`.
```
import numpy as np
from sklearn import svm

N = 50000
trgX = np.random.rand(N,1)
svc = svm.SVC(cache_size=2000)
trgY = np.sign(np.random.sample(N)-0.5)
svc.fit(trgX,trgY)

```


#### Expected Results
Setting `cache_size`>2000 should lead to a corresponding increase in RAM use, and, potentially, a speed up in training time.

#### Actual Results
Setting `cache_size`>2000 leads to very low RAM usage, and no speed up.

#### Versions
Windows-8.1-6.3.9600-SP0
Python 3.5.2 |Anaconda custom (64-bit)| (default, Jul  5 2016, 11:41:13) [MSC v.1900 64 bit (AMD64)]
NumPy 1.11.1
SciPy 0.18.0
Scikit-Learn 0.17.1

user: JonathanTay
issue_8008: state: open
title: [BUG] LabelPropagation should use undirected graphs for knn kernel.
body: This is with reference to the PR #6727 and is discussed in the comments in #5774.

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->

The current implementation of `LabelPropagation`, in `semi_supervised` module, creates [directed k-NN graphs](https://github.com/scikit-learn/scikit-learn/blob/78a674875e30417afe9621346ba022834c138351/sklearn/semi_supervised/label_propagation.py#L137) when the `knn` kernel is chosen.

One way to fix it is as shown in [musically-ut/semi_supervised](https://github.com/musically-ut/semi_supervised/blob/master/semi_supervised/label_propagation.py#L136).

#### ~Steps to Reproduce / Expected Results / Actual Results~

This is part of the reason I didn't submit it as a separate issue: it is difficult to come up with test cases which are _obviously_ correct or incorrect.

However, none of the papers about semi-supervised learning cited in #6727 considers directed graphs. Using directed graphs has some obvious problems, like rendering parts of the graph completely unreachable, thus limiting the _propagation_ of labels only from roots to descendants.

user: musically-ut
issue_7996: state: open
title: [MRG] Tests for no sparse y support in RandomForests
body: #### Reference Issue
Fixes #7886 


#### What does this implement/fix? Explain your changes.
This adds the test that we don't support sparse `y` as of yet for multilabel estimators.

#### Any other comments?
Since `RandomTreesEmbedding` doesn't make use of `y`, so I was not really clear as to how should it be included in the tests and hence, have left it out currently. Also, I did cross the maximum characters limit for the sake of clarity by a very small margin and noticed that there are other such lines (bigger)  present, hence, chose to stick to it.


user: dalmia
issue_7977: state: open
title: incremental update dbscan
body: is there any plan to add incremental clustering for sklearn.cluster.DBSCAN ?
user: manuel-delverme
issue_7975: state: open
title: A possible alternative to alpha at fit-time for GaussianProcessRegressor
body: I love the new GaussianProcessRegressor class, thanks so much.  One thing that I think would make it a little easier for me to use/explain to others is if there were an alternative to the `alpha` option in the constructor to include the "weight" of individual samples.  Here is a minimal example of the current approach:
```
import numpy as np, sklearn.gaussian_process, sklearn.gaussian_process.kernels

X = [[0], [1]]
y = [.5, .5]
se = np.array([.1, .5])

kernel = sklearn.gaussian_process.kernels.Matern(length_scale=1.0, nu=1.5)
gp = sklearn.gaussian_process.GaussianProcessRegressor(kernel=kernel, alpha=se**2, optimizer=None) # set data variance with alpha parameter _here_
gp.fit(X, y)  # then set data values _here_
```
The second row of data has 5x more variation than the first, and the GPR handles this beautifully ([see notebook](https://gist.github.com/aflaxman/f6ea10a6ab1e72a7bebe608883022e56)), but I find it aesthetically unappealing to set `alpha`  in the constructor when the data that needs to match it is not set until the `fit` method is called.

I would prefer an approach where the alpha value was set in the fit function, such as
```
gp = sklearn.gaussian_process.GaussianProcessRegressor(kernel=kernel, optimizer=None)
gp.fit(X, y, alpha=se**2)
```
or, since it seems like `sample_weight` is used for doing this in `LinearRegression` and other places, it might be more consistent to use that instead of `alpha`:
```
gp.fit(X, y, sample_weight=se**-2)
```

I can potentially put together a pull request if this is a change that you are interested in.  Thanks again!
user: aflaxman
issue_7973: state: open
title: Determining parameter behavior based on float / int type
body: There are many places in scikit-learn where ints and floats (usually between 0 and 1) are allowed, and have different semantics.
Most of us think this is brittle. This is an issue to discuss whether we should deprecate and use a different interface, and what that interface should be.

I think the main issue arises when people pass 1.0 vs 1 and 0.0 vs 0 (though they are likely to have the same semantics).
I think we can keep the current semantics (as they are very concise) as long as we are careful about 1.0 and 1.

@GaelVaroquaux proposed to use a more explicit syntax, like "10%", which is a non-backward-compatible change and needs a full deprecation of all floats 0-1. There is a question on whether this applies to all floats between 0 and 1 or only those where also an integer is possible.

My half-baked idea would be to replace 1.0 by "all", which is quite intuitive. I'm not sure what to do about 1, though. We could require something like "one", which would be a bit inconsistent, but solve all use-cases (grid-searching from 1 to 10 will be odd, though).

However, if we do the ``%``, then grid-searching fractions becomes pretty ugly.

user: amueller
labels: Needs Decision: Requires decision
issue_7925: state: open
title: Cross-validation without copying
body: At the moment cross-validation requires indexing an array (or sparse matrix or dataframe) in a way that copies the selected subset of the data. If memory is an issue relative to computation (particular for data being processed in parallel), we could consider modes in which:
* the cross-validation samples are merely weightings over the original dataset (weight=1 where the instance is included in the sample, 0 if excluded), rather than extractions from it.
* `partial_fit` is called over contiguous subsets of the data. Apart from only applying to some estimators, this is complicated by the need to specify multiple iterations with `partial_fit`.

thoughts?
user: jnothman
issue_7916: state: open
title: [WIP] Gaussian Bernoulli RBM continuation of #2680
body: It would be nice to have this feature so I decided to finish #2680.
To-do:
- [X] Use BaseRBM for both BernoulliRBM and GaussianBernoulliRBM
- [ ] Test Gaussian Bernoulli RBM
- [ ] Illustrative example
user: yl565
issue_7910: state: open
title: [MRG+1] Support Vector Data Description
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
This PR covers previous issues concerning SVDD:
* the original unfinished PR #3201 which is now very outdated;
* Issue #2807 requesting SVDD to be added to Scikit;
* PR #5899 which attempted to revive PR #3201 but was closed;
* Unfinished PR #3013 which tried to modify the libsvm part only.

#### What does this implement/fix? Explain your changes.
This PR offers the following:
* SVDD-L1 implementation based on this [libsvm implementation](http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/#libsvm_for_svdd_and_finding_the_smallest_sphere_containing_all_data). The model was extended to the case of a penalty cost vector (in line with other implemented support vector models in Scikit);
* documentation, outlining the SVDD-L1 model;
* an example showing a difference between the One-Class SVM and SVDD models.

#### Any other comments?
The original model was proposed by Tax and Duin (2004), and later reformulated by Chang, Lee, Lin (2013). This PR implements the latter reformulation and extends it to the case of different weights of the observations. I can provide proof of the key theorems if necessary.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: ivannz
issue_7907: state: open
title: `transform` method of tree ensembles should return the `decision_path`
body: Tree ensmbles now have a new `decision_path` method that rturns a boolean matrix indicating which item falls under which node. At the same time the existing `transform` method on these estimators is being deprecated.

The decision path is a very usefu set ofl fetures and would be a ntural `transform` for these estimators,
once the original `transform` is removed.
In particular it would make it very simple to implement Jerome Friedman's _Imprtance Sampled Learning Ensembles._

user: mhlr
issue_7886: state: open
title: Test support for sparse multilabel format
body: Mutlilabel data is indicated by having `y` be a 2d array consisting of 0s and 1s. This may be represented with a sparse matrix format. However, it seems we are lacking in tests of sparse representations of multilabel formats.

We'd like to see one or both of:
* specific tests for each multilabel estimator (or confirmation that they already exist)
* a general test for multilabel estimators that they support sparse `y` and behave identically to dense `y`
* clarity in the estimator docstrings where sparse `y` is supported.
user: jnothman
issue_7834: state: open
title: Is there an issue with TimeSeriesSplit and the learning_curve?
body: #### Description
When using the TimeSeriesSplit (TSS) CV generator with learning_curve(), the learning curve training sets appear to be computed as fractions of the first CV split, as opposed to the whole data set. 

This means that the learning curve is being computed on the basis of a very small training set (just a small fraction of the total training set).

#### Steps/Code to Reproduce

```
from sklearn.datasets import make_classification
from sklearn.model_selection import TimeSeriesSplit
from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
import numpy as np

data_set = make_classification(n_samples=2000, n_features=20, n_informative=2, 
                               n_redundant=2, n_repeated=0, n_classes=2, 
                               n_clusters_per_class=2, weights=None, flip_y=0.01, 
                               class_sep=1.0, hypercube=True, shift=0.0, scale=1.0, 
                               shuffle=True, random_state=None)

X, y = data_set

tscv = TimeSeriesSplit(n_splits=10)

estimator = DecisionTreeClassifier()

train_sizes, train_scores, valid_scores = learning_curve(estimator, X, y, 
                                                         train_sizes=(np.linspace(0.1, 1.0, num=10)), 
                                                         cv=tscv, scoring='f1', exploit_incremental_learning=False, 
                                                         n_jobs=-1, pre_dispatch='2*n_jobs', verbose=1)

```

#### Expected Results
learning curve output looks like this: 
`[learning_curve] Training set sizes: [ 181  363  545  727  909 1091 1273 1455 1637 1819]`

#### Actual Results
Training set sizes are calculated as a fraction of the smallest CV iteration - i.e. a fraction of 181, meaning that the learning curve ends up being very small. 

#### Possible fix
It is possible that line 750 in _validation.py needs to change from
`n_max_training_samples = len(cv_iter[0][0])`
To
`n_max_training_samples = len(cv_iter[-1][0])`

This would allow the learning curve to be computed on basis of the full size of the CV sets, rather than just the first (and smallest) fraction.

#### Versions
Windows-10-10.0.14393
('Python', '2.7.12 |Continuum Analytics, Inc.| (default, Jun 29 2016, 11:07:13) [MSC v.1500 64 bit (AMD64)]')
('NumPy', '1.11.2')
('SciPy', '0.18.1')
('Scikit-Learn', '0.18')

user: indianajensen
issue_7829: state: open
title: Add Angular distance, a metric version of cosine distance
body: Many algorithms, such as word2vec result in nearest neighbor computations based on cosine similarity. Unfortunately, since cosine (dis)similarity is not a metric it can't be used with kd-trees and ball-trees. This means that algorithms that make use of these structures (e.g. DBSCAN clustering, fast t-SNE, etc.) can't operate with regard to the "appropriate" (dis)similarity measure. Here we add angular (or arccos) distance which is the natural metric analogue of cosine dissimilarity to the valid metrics used for kd-trees and ball-trees. Credit for this work belongs to @brunoalano who submitted a similar change to hdbscan.


user: lmcinnes
labels: Needs Decision: Requires decision
issue_7821: state: open
title: Allow alpha in GaussianProcessRegressor at fit time?
body: #### Description
When alpha is an array, it is a data-dependent property.

user: MechCoder
issue_7774: state: open
title: Changing the label order in export_graphviz for interpretability
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->
#### Description

<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->

I occasionally present results to experimental folks, who really love decision tree models due to their interpretability. For informal meetings, I often use the `export_graphviz` function to create the  decision tree viz and talk them through the results. However, I noticed that they are often confused about the labels in the node. I.e., what's in that node vs. what's the splitting criterion. "Do these samples at this node have a feature value x <= y?", "no, this would be the splitting criterion to produce the following child nodes"

Although, it should be somewhat clear from looking at the first node that the first line with the feature name refers to the splitting criterion, maybe it would be worthwhile swapping the order? This way the statistics in the node apply to the samples in this actual node, and the last line (the splitting criterion) would refer to the two child nodes going off from the parent node. 

Or in simpler words, what do you think about putting the splitting criterion at the bottom of a node?

![screen shot 2016-10-27 at 7 28 27 pm](https://cloud.githubusercontent.com/assets/5618407/19789495/15424754-9c7e-11e6-9909-b3e01f24d556.png)

And maybe even putting a divider line in-between?

![screen shot 2016-10-27 at 7 50 35 pm](https://cloud.githubusercontent.com/assets/5618407/19789567/a93dbbaa-9c7e-11e6-8b79-36e64e0dbc05.png)

user: rasbt
issue_7773: state: open
title: [MRG+½] label_binarize also for binary case
body: #### Reference Issue #6191
#### What does this implement/fix? Explain your changes.

changes in function label_binarize(...):
- extended doc string
- change pos and neg labels just to be non equal
- added param to force return matrix even for binary case
- minor code cleaning (rename Y -> y_out according pep8)

## Checklist
- [x] Clean style in [the spirit of PEP8](https://www.python.org/dev/peps/pep-0008/)
- [x] [Docstrings for all functions](https://github.com/numpy/numpy/blob/master/doc/example.py)
- [x] Unit tests
user: Borda
labels: Needs Decision: Requires decision
issue_7755: state: open
title: parallel votingClassifier prediction method needed
body: Hi there:
I am using a really big jungle of single classifiers for an ensemble voting, something like:

emclf = VotingClassifier(estimators=est_list,n_jobs=-1)
the est_list has about 300 classifiers.
---->that's all good and thank you folks.

However, when I am done training and doing predictions, like : 

prd = emclf.predict(test_x)

**the prediction method is not conducted in a parallel way, which became a bottleneck for the performance.**

Is that possible to make this predict step parallel? thanks

user: kingfengji
issue_7754: state: open
title: enhancement: sklearn.utils.shuffle consume 2x memory, better do it in-place
body: <!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->
#### Description

<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->

sklearn.utils.shuffle use the double amount of memory in my test. I'd like the in-place implementation without using 2x times memory of the input arrays.

https://github.com/numpy/numpy/issues/8204
#### Steps/Code to Reproduce

<!--
Example:
```
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->
#### Expected Results

<!-- Example: No error is thrown. Please paste or describe the expected results.-->
#### Actual Results

<!-- Please paste or specifically describe the actual output or traceback. -->
#### Versions

<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->

<!-- Thanks for contributing! -->

user: mw66
issue_7722: state: open
title: permutation_test_score does not handle output from GridSearchCV
body: Suppose that you wish to tune the hyperparameters of a SVC model.

``` python
import numpy as np
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV,StratifiedKFold,permutation_test_score

# get data
feat=np.random.normal(size=(100,2))
lbl=np.random.binomial(1,.5,size=(100,))
wghts=np.ones(shape=(100,))

# fit model
pipe_svc=Pipeline([('scl',StandardScaler()),
                       ('clf',SVC(probability=True))])
param_range=[.1,1.,10]
param_grid=[{'clf__C':param_range,'clf__kernel':['linear']},
            {'clf__C':param_range,'clf__kernel':['rbf'],'clf__gamma':param_range}]
gs=GridSearchCV(estimator=pipe_svc,param_grid=param_grid,scoring='accuracy',
                fit_params={'clf__sample_weight':wghts},cv=10,n_jobs=1)
gs=gs.fit(feat,lbl)
print gs.best_score_,gs.best_params_
```

Now, we would like to compute the test score on the identified gs object.

``` python
cv=StratifiedKFold(n_splits=10)
score,perm_scores,pValue=permutation_test_score(estimator=gs,X=feat,
    y=lbl,cv=cv,n_permutations=100,n_jobs=1,scoring='accuracy')
```

The following error appears:

``` python
ValueError: sample_weight and X have incompatible shapes: (100L,) vs (79L, 2L)
Note: Sparse matrices cannot be indexed w/boolean masks (use `indices=True` in CV).
```

Versions: numpy 1.10.4, sklearn 0.18

user: JohnNapier
issue_7686: state: open
title: Update ml_map.png
body: As we have deprecated the GMM and VBGMM classes we must modify the [`ml_map.png`](http://scikit-learn.org/stable/tutorial/machine_learning_map/).
To modify it correctly, you know where I can find the original file (gimp file, ...) ? 

user: tguillemot
labels: Easy: Well-defined and straightforward way to resolve
issue_7608: state: open
title: RFC binary operators on estimators
body: So maybe this is not that bad an idea?

https://gist.github.com/amueller/643f812a275a9e0c75048aab6988a92c

@jnothman can you maybe point me to what you meant?

user: amueller
labels: Needs Decision - API: Requires decision regarding API
issue_7574: state: open
title: [Feature Request] Progress Bars
body: I'm currently working with a fork of sklearn where I include progress bars in long running Estimators (particularly MiniBatchKmeans). It takes up to 9 hours to run some of the algorithms and having an estimated time remaining has been quite helpful to see when I should come back and look at the results.

I was wondering if it would be worthwhile to compile some of these into a pull requests and add this functionality to the library. If so there are few logistics: 

Which progress implementation should I use? I see three options here.

   1)  I could add a dependency on the external click library and use its ProgressBar class. 
   2) I could port my a simplified version of my own ProgIter class into sklearn utils.  
   3) I could port a simplified version of click's ProgressBar class 

All options essentially wrap an iterator and dump some info to stdout about how far through the iterator they are. 

These options are ordered from the least work to the most work. The first option is the least work, but it adds a dependency. I have experience with the second option, but it is not as widely used as click (although my port would use a more click-like signature). The third option requires me to delve into the click library, which I haven't had any experience with yet. However, judging from my initial look at its implementation it looks my implementation may be more efficient. The click implementation seems to compute and display progress information on every iteration, whereas my implementation tries to minimally impact the wrapped loop by adjusting its report frequency. 

Here is some example code. Previously a code block looked like this: 

``` python
        if self.verbose:
            print('Computing label assignment and total inertia')
        x_squared_norms = row_norms(X, squared=True)
        slices = gen_batches(X.shape[0], self.batch_size)
        results = [_labels_inertia(X[s], x_squared_norms[s],
                                   self.cluster_centers_) for s in slices]
        labels, inertia = zip(*results)
        return np.hstack(labels), np.sum(inertia)
```

If I port a refactored / paired down version of my progress iterator to sklearn, then it will look something like this: 

``` python
        if self.verbose:
            print('Computing label assignment and total inertia')
        x_squared_norms = row_norms(X, squared=True)
        n_samples = X.shape[0]
        total_batches = int(n_samples // self.batch_size)
        slices = gen_batches(n_samples, self.batch_size)
        slices = ProgIter(slices, label='computing labels and inertia',
                          length=total_batches, enabled=self.verbose)
        results = [_labels_inertia(X[s], x_squared_norms[s],
                                   self.cluster_centers_) for s in slices]
        labels, inertia = zip(*results)
        return np.hstack(labels), np.sum(inertia)
```

Here is a mockup of the simplified ProgIter object. I paired it down rather quickly, so there may be some small issue in this version. 

``` python
from __future__ import absolute_import, division, print_function, unicode_literals
import sys
import time
import datetime
import collections
import six

WIN32 = sys.platform.startswith('win32')

if WIN32:
    # Use time.clock in win32
    default_timer = time.clock
    BEFORE_BAR = '\r'
    AFTER_BAR = '\n'
else:
    default_timer = time.time
    BEFORE_BAR = '\r\033[?25l'
    AFTER_BAR = '\033[?25h\n'

text_type = six.text_type

class ProgIter(object):
    def __init__(self, iterable=None, label=None, length=None, enabled=True,
                 freq=1, adjust=True, eta_window=64):
        if length is None:
            try:
                length = len(iterable)
            except Exception:
                pass
        if label is None:
            label = ''
        self.iterable = iterable
        self.label = label
        self.length = length
        self.freq = freq
        self.enabled = enabled
        self.adjust = adjust
        self.count = 0
        # Window size for estimated time remaining
        self.eta_window = eta_window

    def __call__(self, iterable):
        self.iterable = iterable
        return self

    def __iter__(self):
        if not self.enabled:
            return iter(self.iterable)
        else:
            return self.iter_rate()

    def build_msg_fmtstr(self):
        with_wall = True
        tzname = time.tzname[0]
        length_ = '?' if self.length == 0 else text_type(self.length)
        msg_head = ['', self.label, ' {count:4d}/', length_ , '...  ']
        msg_tail = [
            ('rate={rate:4.2f} Hz,'),
            ('' if self.length == 0 else ' eta={eta},'),
            ' ellapsed={ellapsed},',
            (' wall={wall} ' + tzname if with_wall else ''),
        ]
        msg_fmtstr_time = ''.join(([BEFORE_BAR] + msg_head + msg_tail))
        return msg_fmtstr_time

    def tryflush(self):
        try:
            # flush sometimes causes issues in IPython notebooks
            sys.stdout.flush()
        except IOError:
            pass

    def write(self, msg):
        sys.stdout.write(msg)

    def iter_rate(self):
        freq          = self.freq
        self.count    = 0
        between_count = 0
        last_count    = 0

        # how long iterations should be before a flush
        # (used for freq adjustment)
        time_thresh = 2.0

        max_between_time = -1.0
        max_between_count = -1.0

        iters_per_second = -1

        # Prepare for iteration
        msg_fmtstr = self.build_msg_fmtstr()

        self.tryflush()

        msg = msg_fmtstr.format(
            count=self.count, rate=0.0,
            eta=text_type('0:00:00'),
            ellapsed=text_type('0:00:00'),
            wall=time.strftime('%H:%M'),
        )
        self.write(msg)

        self.tryflush()

        start_time = default_timer()
        last_time  = start_time

        # use last few (64) times to compute a more stable average rate
        measure_between_time = collections.deque([], maxlen=self.eta_window)
        measure_est_seconds = collections.deque([], maxlen=self.eta_window)

        # Wrap the for loop with a generator
        for self.count, item in enumerate(self.iterable, start=1):
            yield item

            if (self.count) % freq == 0:
                # update progress information every so often
                now_time          = default_timer()
                between_time      = (now_time - last_time)
                between_count     = self.count - last_count
                total_seconds     = (now_time - start_time)

                measure_between_time.append(between_count / (float(between_time) + 1E-9))
                iters_per_second = sum(measure_between_time) / len(measure_between_time)

                if self.length is None:
                    est_seconds_left = -1
                else:
                    iters_left = self.length - self.count
                    est_eta = iters_left / (iters_per_second + 1E-9)

                    measure_est_seconds.append(est_eta)
                    est_seconds_left = sum(measure_est_seconds) / len(measure_est_seconds)

                last_count = self.count
                last_time  = now_time
                # Adjust frequency if printing too quickly
                # so progress doesnt slow down actual function
                if self.adjust and (between_time < time_thresh or
                                    between_time > time_thresh * 2.0):
                    max_between_time = max(max_between_time, between_time)
                    max_between_time = max(max_between_time, 1E-9)
                    max_between_count = max(max_between_count, between_count)
                    # If progress was uniform and all time estimates were
                    # perfect this would be the new freq to achieve time_thresh
                    new_freq = int(time_thresh * max_between_count / max_between_time)
                    new_freq = max(new_freq, 1)
                    # But things are not perfect. So, don't make drastic changes
                    max_freq_change_up = max(256, freq * 2)
                    max_freq_change_down = freq // 2
                    if (new_freq - freq) > max_freq_change_up:
                        freq += max_freq_change_up
                    elif (freq - new_freq) > max_freq_change_down:
                        freq -= max_freq_change_down
                    else:
                        freq = new_freq
                msg = msg_fmtstr.format(
                    count=self.count,
                    rate=iters_per_second,
                    eta=text_type(datetime.timedelta(seconds=int(est_seconds_left))),
                    ellapsed=text_type(datetime.timedelta(seconds=int(total_seconds))),
                    wall=time.strftime('%H:%M'),
                )
                self.write(msg)
                self.tryflush()

        if (self.count) % freq != 0:
            # Write the final progress line if it was not written in the loop
            est_seconds_left = 0
            now_time = default_timer()
            total_seconds = (now_time - start_time)
            msg = msg_fmtstr.format(
                count=self.count,
                rate=iters_per_second,
                eta=text_type(datetime.timedelta(seconds=int(est_seconds_left))),
                ellapsed=text_type(datetime.timedelta(seconds=int(total_seconds))),
                wall=time.strftime('%H:%M'),
            )
            self.write(msg)
            self.tryflush()
        self.write(AFTER_BAR)
```

Here is an example showing what this does:

``` python
# Define a function that takes some time
def is_prime(n):
    return n >= 2 and not any(n % i == 0 for i in range(2, n))
N = 50000

# Default behavior adjusts frequency of progress reporting so
# the performance of the loop is minimally impacted
iterable = (is_prime(n) for n in range(N))
piterable = ProgIter(iterable, length=N)
_ = list(piterable)
 50000/50000...  rate=25097.04 Hz, eta=0:00:00, ellapsed=0:00:22, wall=11:22 EST

# Adjustments can be turned off to give constant feedback
iterable = (is_prime(n) for n in range(N))
piterable = ProgIter(iterable, length=N, adjust=False, freq=100)
_ = list(piterable)
  2200/50000...  rate=32500.44 Hz, eta=0:00:01, ellapsed=0:00:00, wall=11:23 EST
43300/50000...    rate=1398.73 Hz, eta=0:00:06, ellapsed=0:00:18, wall=11:23 EST
50000/50000...  rate=1129.99 Hz, eta=0:00:02, ellapsed=0:00:25, wall=11:25 EST

# only one line of the above output is shown at a time, I copied different points from a few runs 
# to illustrate the output. 
```

Lastly here is some timeit information that shows how frequency adjusting causes minimal overhead

``` python
def is_prime(n):
    return n >= 2 and not any(n % i == 0 for i in range(2, n))

N = 1000

# Test time of raw loop without any fanciness 
%timeit list((is_prime(n) for n in range(N)))
100 loops, best of 3: 10 ms per loop

# Test time of raw loop when printing on every iteration
%timeit list(ProgIter((is_prime(n) for n in range(N)), length=N, freq=1, adjust=False))
10 loops, best of 3: 159 ms per loop

# Test time of raw loop when printing based on an adjusted schedule 
%timeit list(ProgIter((is_prime(n) for n in range(N)), length=N, freq=1, adjust=True))
100 loops, best of 3: 11 ms per loop


```

user: Erotemic
issue_7559: state: open
title: Tolerance on RFECV
body: Hi,

Is there any way to specify a tolerance when determining the number of optimal features when using RFECV (like n the CARET package)? Currently when using RFECV with many of the tree-based classifiers, the removal of the least important n features often results in a very slight reduction in model accuracy. This means that all of the features end up being selected as 'optimal', even though the reduction in model accuracy is very slight if we used a much smaller set of features. A tolerance setting, like 1% would allow only the features to be selected that would otherwise cause a large drop in model accuracy.

Steve

user: stevenpawley
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_7549: state: open
title: TfIdf vectorizer partial_fit
body: Couldn't something like this be absorbed into the module?

http://stackoverflow.com/a/39114555/4088570

user: gpkc
issue_7529: state: open
title: Are there any plans to implement Labelled LDA?
body: Labelled LDA is a "topic-model"-ish algorithm similar to LDA, except that it's supervised. You give it some labels and documents with those labels, and it models the words that are in each of the labels. Does this algorithm exist currently in scikit-learn? If not, are there any current plans to implement it?

user: ghost
issue_7492: state: open
title: KernelDensity Score unstable with many samples and changing leaf_size
body: Hi,

I have the impression that KernelDensity.score_samples() behaves strangely when there are many samples to fit.  
#### Description

When I fit many samples adding a single additional sample has a very large impact on the score of an unchanged test set. I think the influence of a single additional training input should become smaller rather than larger, as the number of training samples goes up (as can be seen below this is initially the case, before strange behaviour occurs at >80 training samples). 

I get the expected behaviour when I implement the KDE by hand.
#### Code

Here's the code to reproduce the behaviour:

``` py
import numpy as np
from sklearn.neighbors import KernelDensity as KD
from sklearn.datasets import load_digits

###Load the digits dataset and split it into a training and a testing set                                                                                                                                            

trainSamp = load_digits()['images']
trainSampF = np.zeros((trainSamp.shape[0],trainSamp.shape[1]*trainSamp.shape[2]))
for i in range(trainSamp.shape[0]):
     trainSampF[i] = trainSamp[i].flatten()

testSamp = trainSampF[-100:]
trainSamp = trainSampF[:-100]
train = trainSamp[:]
test = testSamp[:20]

###Fit a gaussian KDE with k*10 samples and score                                                                                                                                                                    
###compare to a gaussian KDE with one additional sample and score                                                                                                                                                   

for k in range(10):
     n = (k+1)*10
     model = KD(bandwidth=0.175, kernel='gaussian')
     model.fit(train[:n])
     llsv = model.score_samples(test)
     model.fit(train[:(n+1)])
     llsv2 = model.score_samples(test)
     print('evaluating score (noise free) for')
     print(n,'training samples',llsv.mean(), ' and',n+1, 'training samples',llsv2.mean())
     print('difference:',llsv.mean()-llsv2.mean())

     print()
```
#### Expected Result

Decreasing impact of adding a single training sample as training set gets larger.
#### Actual Result

When there are more than ~80 training samples adding a single training sample changes the score quite a lot.

Program output:

```
evaluating score (noise free) for
10 training samples -16898.9523708  and 11 training samples -16779.047681
difference: -119.90468982

evaluating score (noise free) for
20 training samples -15013.1149057  and 21 training samples -15013.1636959
difference: 0.0487901641663

evaluating score (noise free) for
30 training samples -14134.3366974  and 31 training samples -14134.3694872
difference: 0.0327898228261

evaluating score (noise free) for
40 training samples -11150.9509101  and 41 training samples -11150.9756027
difference: 0.0246926125947

evaluating score (noise free) for
50 training samples -11148.725074  and 51 training samples -11148.7448766
difference: 0.0198026272956

evaluating score (noise free) for
60 training samples -10508.091069  and 61 training samples -10508.1075983
difference: 0.0165293019545

evaluating score (noise free) for
70 training samples -10508.2452197  and 71 training samples -10508.2594044
difference: 0.0141846349907

evaluating score (noise free) for
80 training samples -10024.2971185  and 81 training samples -10082.2179183
difference: 57.9207998393

evaluating score (noise free) for
90 training samples -9293.12908873  and 91 training samples -9565.75291888
difference: 272.623830155

evaluating score (noise free) for
100 training samples -9855.60678148  and 101 training samples -10123.3057225
difference: 267.698941066
```
#### Versions

Linux-3.19.0-66-generic-x86_64-with-debian-jessie-sid
('Python', '2.7.12 |Anaconda 4.0.0 (64-bit)| (default, Jul  2 2016, 17:42:40) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]')
('NumPy', '1.11.1')
('SciPy', '0.17.1')
('Scikit-Learn', '0.17.1')
#### What now?

I would be glad if you could help me figure out what I did wrong or what's going wrong. Thanks!

<!-- Thanks for contributing! -->

user: lorenzMuller
labels: Moderate: Anything that requires some knowledge of conventions and best practices
Needs Investigation: Issue requires investigation
issue_7489: state: open
title: Infinite loop or never returns for logistic regression in nearly degenerate case
body: #### Description

Logistic Regression never returns on fitting with nearly degenerate data.
#### Steps/Code to Reproduce

``` python
import sklearn.linear_model
import numpy as np
model = sklearn.linear_model.LogisticRegression()
num_pts = 15
x = np.zeros((num_pts*2, 2))
x[3] = 3.7491010398553741e-208
y = np.append(np.zeros(num_pts), np.ones(num_pts))
model.fit(x, y)
```
#### Expected Results

Return or throw error.
#### Actual Results

Never returns.
#### Versions

Linux-2.6.32-573.18.1.el6.x86_64-x86_64-with-redhat-6.7-Carbon
('Python', '2.7.12 |Anaconda 2.0.1 (64-bit)| (default, Jul  2 2016, 17:42:40) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]')
('NumPy', '1.11.0')
('SciPy', '0.17.0')
('Scikit-Learn', '0.17.1')

user: MarvinT
issue_7486: state: open
title: LogisticRegression freezed when X containing large number
body: The following code will freeze

``` python
X = np.array([0, 1e+100]).reshape(-1, 1)
y = np.array([0, 1])
clf = LogisticRegression(solver='liblinear').fit(X, y)
```

It seems to be a liblinear problem, the while loop [here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/src/liblinear/tron.cpp#L170) is not breaking.

Maybe a warning could be shown suggesting user to standardize data...

user: yl565
issue_7479: state: open
title: Support a sparse coef_ in SelectFromModel
body: Currently it looks like we only support a numpy array as an estimator's `coef_` in `SelectFromModel`. We should also be supporting  `coef_` as `scipy.sparse.csr_matrix` (note that linear models have a `sparsify` function that creates such a thing, so in general when we test behaviour with `coef_` we should allow for it to be a sparse matrix).

user: jnothman
issue_7478: state: open
title: SelectFromModel.transform is very slow: > 0.25 seconds to run on a single row
body: ### Description

When profiling unexpectedly slow production code, I was shocked to find out that `SelectFromModel.transform()` took over a quarter of a second to run on a single row, while everything else in the Pipeline took very roughly a microsecond to run on a single row. 

We're using this in production with private data, so I can't copy exactly the example we're using. But the dataset that we feed into `SelectFromModel.fit` has many hundreds of data points, while the pruned version of the dataset after feature selection has only dozens of data points. 

The compute time required for `SelectFromModel.transform()` over tens of thousands of rows is only a fraction of a second longer than the compute time required for a single row. We can probably pre-calculate a lot of what `SelectFromModel.transform` is calculating on the fly each time. 

At some point, I'll dive into the code a little more and submit a PR to speed this up. This slowness would have prevented us from using a pipeline built using scikit-learn in production. Everything else runs very quickly (that's one of the reasons I like scikit-learn so much: a very active community of developers optimizing the performance all the time), but this one bottleneck took orders of magnitude longer than all the other calculations combined when getting a prediction on a single row. 
#### Versions

```
>>> import platform; print(platform.platform())
Darwin-15.6.0-x86_64-i386-64bit
>>> import sys; print("Python", sys.version)
('Python', '2.7.12 (default, Jun 29 2016, 14:05:02) \n[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)]')
>>> import numpy; print("NumPy", numpy.__version__)
('NumPy', '1.11.1')
>>> import scipy; print("SciPy", scipy.__version__)
('SciPy', '0.18.0')
>>> import sklearn; print("Scikit-Learn", sklearn.__version__)
('Scikit-Learn', '0.17.1')
```

Thanks, as always, for running such an awesome project! Hopefully this will help speed up other peoples' production code as well, and encourage even wider adoption of scikit-learn. It seems a shame to have such a highly optimized library bottlenecked by such a primitive piece of functionality. 

user: ClimbsRocks
issue_7475: state: open
title: Why normalize whitespaces for CountVectorizer(analyzer='char')?
body: I am curious of the rational of replacing consecutive whitespaces with just a single space character for [`CountVectorizer(analyzer='char')`](https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/feature_extraction/text.py). I would think it is safe to assume that users who want character ngram counts would want the counts of all characters without any modification.

This is especially problematic if the user wanted ngram counts for n > 1 as consecutive whitespace characters from the original document will not appear due to the normalization process. It may be less of a problem if whitespace normalization is done by replacing every individual whitespace with space characters as consecutive space character ngrams will still be counted.

I can understand the rational for normalizing whitespaces if the `analyzer` argument is set to `'char_wb'` as the white space is used as word delimiters. Even so, I don't think it is necessary to replace all consecutive whitespaces as just a single space character.

I am specifically referring to lines [142-153](https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/feature_extraction/text.py#L142-L153) in the sklearn.feature_extraction.text.py.

```
    def _char_ngrams(self, text_document):
        """Tokenize text_document into a sequence of character n-grams"""
        # normalize white spaces
        text_document = self._white_spaces.sub(" ", text_document)

        text_len = len(text_document)
        ngrams = []
        min_n, max_n = self.ngram_range
        for n in xrange(min_n, min(max_n + 1, text_len + 1)):
            for i in xrange(text_len - n + 1):
                ngrams.append(text_document[i: i + n])
        return ngrams
```

`self._white_spaces` is [defined earlier](https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/feature_extraction/text.py#L100) as `re.compile(r"\s\s+")`.

Hope it is clear.

EDIT: I propose to either remove the white space normalizing step or replace the `_white_spaces` regex as `re.compile(r"\s")` so that all white spaces, even though normalized, are counted.

user: yxtay
issue_7471: state: open
title: Adding Yinyang K-Means implementation
body: The proposed Yinyang K-Means algorithm modification offers a significant speedup while keeping giving exactly the same results (compared to regular K-Means).

Here’s the original paper presented at ICML’15: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ding15.pdf

@amueller kindly informed me that @MechCoder has mentioned it somewhere and probably has started implementation.

@MechCoder, can you please give an update?

user: kirillbobyrev
issue_7383: state: open
title: [MRG] Added flag to disable l2-dist finite check
body: The following change significantly speeds up the kmeans++ initialization used
in MiniBatchKmeans.

The Euclidean distance computation is the bottleneck in kmeans++. However, on
every call to Euclidean distance there is also a call to check_pairwise_arrays. 
In in kmeans++, the same Y arrays are being checked every time. One of the checks 
in here turns out to cause a speed issue. Specifically the finite check. This patch adds a flag to disable this check.

I'm not sure if this is the desired way to go about this change, but I do think something needs to be 
done about this functions efficiency. 

Here is some data that shows the speed increase:
For these tests I'm clustering with  n_clusters=1000. The feature dimension is 128 and 
the number of data points is 10*n_clusters. I then profiled different versions of the code.
First here is the slow version where I force it to perform the finite check every time.

``` python
Total time: 5.67384 s
File: /home/joncrall/code/scikit-learn/sklearn/metrics/pairwise.py
Function: euclidean_distances at line 173

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   173                                           @ut.profile
   174                                           def euclidean_distances(X, Y=None, Y_norm_squared=None, squared=False,
   175                                                                   X_norm_squared=None, force_all_finite=True):
   237      1000      1402395   1402.4     24.7      X, Y = check_pairwise_arrays(X, Y, force_all_finite=True)
   239                                           
   240      1000         1000      1.0      0.0      if X_norm_squared is not None:
   241                                                   XX = check_array(X_norm_squared)
   242                                                   if XX.shape == (1, X.shape[0]):
   243                                                       XX = XX.T
   244                                                   elif XX.shape != (X.shape[0], 1):
   245                                                       raise ValueError(
   246                                                           "Incompatible dimensions for X and X_norm_squared")
   247                                               else:
   248      1000        36429     36.4      0.6          XX = row_norms(X, squared=True)[:, np.newaxis]
   249                                           
   250      1000         1251      1.3      0.0      if X is Y:  # shortcut in the common case euclidean_distances(X, X)
   251                                                   YY = XX.T
   252      1000          976      1.0      0.0      elif Y_norm_squared is not None:
   253      1000        15173     15.2      0.3          YY = np.atleast_2d(Y_norm_squared)
   254                                           
   255      1000         2450      2.5      0.0          if YY.shape != (1, Y.shape[0]):
   256                                                       raise ValueError(
   257                                                           "Incompatible dimensions for Y and Y_norm_squared")
   258                                               else:
   259                                                   YY = row_norms(Y, squared=True)[np.newaxis, :]
   260                                           
   261      1000      3846672   3846.7     67.8      distances = safe_sparse_dot(X, Y.T, dense_output=True)
   262      1000       108011    108.0      1.9      distances *= -2
   263      1000        64575     64.6      1.1      distances += XX
   264      1000        64196     64.2      1.1      distances += YY
   265      1000       128121    128.1      2.3      np.maximum(distances, 0, out=distances)
   266                                           
   267      1000         1648      1.6      0.0      if X is Y:
   268                                                   # Ensure that distances between vectors and themselves are set to 0.0.
   269                                                   # This may not be the case due to floating point rounding errors.
   270                                                   distances.flat[::distances.shape[0] + 1] = 0.0
   271                                           
   272      1000          941      0.9      0.0      return distances if squared else np.sqrt(distances, out=distances)
```

Digging a little deeper shows the timings int his function in this function.
As you can see most of this functions time is spent in `check_array`.

``` python
Total time: 1.40017 s
File: /home/joncrall/code/scikit-learn/sklearn/metrics/pairwise.py
Function: check_pairwise_arrays at line 58

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    58                                           @ut.profile
    59                                           def check_pairwise_arrays(X, Y, precomputed=False, dtype=None,
    60                                                                     force_all_finite=True):
   104      1000         8652      8.7      0.6      X, Y, dtype_float = _return_float_dtype(X, Y)
   105                                           
   106      1000          659      0.7      0.0      warn_on_dtype = dtype is not None
   107      1000          460      0.5      0.0      estimator = 'check_pairwise_arrays'
   108      1000          507      0.5      0.0      if dtype is None:
   109      1000          524      0.5      0.0          dtype = dtype_float
   110                                           
   111      1000          581      0.6      0.0      if Y is X or Y is None:
   112                                                   X = Y = check_array(X, accept_sparse='csr', dtype=dtype,
   113                                                                       warn_on_dtype=warn_on_dtype, estimator=estimator,
   114                                                                       force_all_finite=force_all_finite)
   115                                               else:
   116      1000          761      0.8      0.1          X = check_array(X, accept_sparse='csr', dtype=dtype,
   117      1000          481      0.5      0.0                          warn_on_dtype=warn_on_dtype, estimator=estimator,
   118      1000       153559    153.6     11.0                          force_all_finite=force_all_finite)
   119      1000          633      0.6      0.0          Y = check_array(Y, accept_sparse='csr', dtype=dtype,
   120      1000          514      0.5      0.0                          warn_on_dtype=warn_on_dtype, estimator=estimator,
   121      1000      1230357   1230.4     87.9                          force_all_finite=force_all_finite)
   122                                           
   123      1000          559      0.6      0.0      if precomputed:
   124                                                   if X.shape[1] != Y.shape[0]:
   125                                                       raise ValueError("Precomputed metric requires shape "
   126                                                                        "(n_queries, n_indexed). Got (%d, %d) "
   127                                                                        "for %d indexed." %
   128                                                                        (X.shape[0], X.shape[1], Y.shape[0]))
   129      1000         1287      1.3      0.1      elif X.shape[1] != Y.shape[1]:
   130                                                   raise ValueError("Incompatible dimension for X and Y matrices: "
   131                                                                    "X.shape[1] == %d while Y.shape[1] == %d" % (
   132                                                                        X.shape[1], Y.shape[1]))
   133                                           
   134      1000          638      0.6      0.0      return X, Y
```

Looking at `check_array` we see the offending function call to `_assert_all_finite`

``` python
Total time: 1.28438 s
File: /home/joncrall/code/scikit-learn/sklearn/utils/validation.py
Function: check_array at line 271

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   271                                           @ut.profile
   272                                           def check_array(array, accept_sparse=None, dtype="numeric", order=None,
   273                                                           copy=False, force_all_finite=True, ensure_2d=True,
   274                                                           allow_nd=False, ensure_min_samples=1, ensure_min_features=1,
   275                                                           warn_on_dtype=False, estimator=None):

   342      2001         2498      1.2      0.2      if isinstance(accept_sparse, str):
   343      2000         2531      1.3      0.2          accept_sparse = [accept_sparse]
   344                                           
   345                                               # store whether originally we wanted numeric dtype
   346      2001         4113      2.1      0.3      dtype_numeric = dtype == "numeric"
   347                                           
   348      2001         3419      1.7      0.3      dtype_orig = getattr(array, "dtype", None)
   349      2001         4128      2.1      0.3      if not hasattr(dtype_orig, 'kind'):
   350                                                   # not a data type (e.g. a column named dtype in a pandas DataFrame)
   351                                                   dtype_orig = None
   352                                           
   353      2001         1811      0.9      0.1      if dtype_numeric:
   354         1            1      1.0      0.0          if dtype_orig is not None and dtype_orig.kind == "O":
   355                                                       # if input is object, convert to float.
   356                                                       dtype = np.float64
   357                                                   else:
   358         1            1      1.0      0.0              dtype = None
   359                                           
   360      2001         4210      2.1      0.3      if isinstance(dtype, (list, tuple)):
   361                                                   if dtype_orig is not None and dtype_orig in dtype:
   362                                                       # no dtype conversion required
   363                                                       dtype = None
   364                                                   else:
   365                                                       # dtype conversion required. Let's select the first element of the
   366                                                       # list of accepted types.
   367                                                       dtype = dtype[0]
   368                                           
   369      2001         1944      1.0      0.2      if estimator is not None:
   370      2000         3129      1.6      0.2          if isinstance(estimator, six.string_types):
   371      2000         1908      1.0      0.1              estimator_name = estimator
   372                                                   else:
   373                                                       estimator_name = estimator.__class__.__name__
   374                                               else:
   375         1            1      1.0      0.0          estimator_name = "Estimator"
   376      2001         3926      2.0      0.3      context = " by %s" % estimator_name if estimator is not None else ""
   377                                           
   378      2001         3753      1.9      0.3      if sp.issparse(array):
   379                                                   array = _ensure_sparse_format(array, accept_sparse, dtype, copy,
   380                                                                                 force_all_finite)
   381                                               else:
   382      2001         8476      4.2      0.7          array = np.array(array, dtype=dtype, order=order, copy=copy)
   383                                           
   384      2001         2034      1.0      0.2          if ensure_2d:
   385      2001         2534      1.3      0.2              if array.ndim == 1:
   386                                                           if ensure_min_samples >= 2:
   387                                                               raise ValueError("%s expects at least 2 samples provided "
   388                                                                                "in a 2 dimensional array-like input"
   389                                                                                % estimator_name)
   390                                                           warnings.warn(
   391                                                               "Passing 1d arrays as data is deprecated in 0.17 and will "
   392                                                               "raise ValueError in 0.19. Reshape your data either using "
   393                                                               "X.reshape(-1, 1) if your data has a single feature or "
   394                                                               "X.reshape(1, -1) if it contains a single sample.",
   395                                                               DeprecationWarning)
   396      2001        16453      8.2      1.3              array = np.atleast_2d(array)
   397                                                       # To ensure that array flags are maintained
   398      2001         3891      1.9      0.3              array = np.array(array, dtype=dtype, order=order, copy=copy)
   399                                           
   400                                                   # make sure we actually converted to numeric:
   401      2001         2012      1.0      0.2          if dtype_numeric and array.dtype.kind == "O":
   402                                                       array = array.astype(np.float64)
   403      2001         2282      1.1      0.2          if not allow_nd and array.ndim >= 3:
   404                                                       raise ValueError("Found array with dim %d. %s expected <= 2."
   405                                                                        % (array.ndim, estimator_name))
   406      2001         1898      0.9      0.1          if force_all_finite:
   407      2001      1114783    557.1     86.8              _assert_all_finite(array)
   408                                           
   409      2001        61351     30.7      4.8      shape_repr = _shape_repr(array.shape)
   410      2001         2295      1.1      0.2      if ensure_min_samples > 0:
   411      2001        16251      8.1      1.3          n_samples = _num_samples(array)
   412      2001         2250      1.1      0.2          if n_samples < ensure_min_samples:
   413                                                       raise ValueError("Found array with %d sample(s) (shape=%s) while a"
   414                                                                        " minimum of %d is required%s."
   415                                                                        % (n_samples, shape_repr, ensure_min_samples,
   416                                                                           context))
   417                                           
   418      2001         2557      1.3      0.2      if ensure_min_features > 0 and array.ndim == 2:
   419      2001         2229      1.1      0.2          n_features = array.shape[1]
   420      2001         1982      1.0      0.2          if n_features < ensure_min_features:
   421                                                       raise ValueError("Found array with %d feature(s) (shape=%s) while"
   422                                                                        " a minimum of %d is required%s."
   423                                                                        % (n_features, shape_repr, ensure_min_features,
   424                                                                           context))
   425                                           
   426      2001         1958      1.0      0.2      if warn_on_dtype and dtype_orig is not None and array.dtype != dtype_orig:
   427                                                   msg = ("Data with input dtype %s was converted to %s%s."
   428                                                          % (dtype_orig, array.dtype, context))
   429                                                   warnings.warn(msg, _DataConversionWarning)
   430      2001         1776      0.9      0.1      return array
```

Disabling this check after it runs the first time gives a better profile.

We could probably even scrape a bit more performance by checking everything at
the start of kmeans++ and then disabling all subsequent checks. This should be
ok because the new centroids are always just previously existing ones. 

```
Total time: 3.82659 s
File: /home/joncrall/code/scikit-learn/sklearn/metrics/pairwise.py
Function: euclidean_distances at line 173

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   173                                           @ut.profile
   174                                           def euclidean_distances(X, Y=None, Y_norm_squared=None, squared=False,
   175                                                                   X_norm_squared=None, force_all_finite=True):
   238      1000       288126    288.1      7.5      X, Y = check_pairwise_arrays(X, Y, force_all_finite=force_all_finite)
   239                                           
   240      1000          718      0.7      0.0      if X_norm_squared is not None:
   241                                                   XX = check_array(X_norm_squared)
   242                                                   if XX.shape == (1, X.shape[0]):
   243                                                       XX = XX.T
   244                                                   elif XX.shape != (X.shape[0], 1):
   245                                                       raise ValueError(
   246                                                           "Incompatible dimensions for X and X_norm_squared")
   247                                               else:
   248      1000        21165     21.2      0.6          XX = row_norms(X, squared=True)[:, np.newaxis]
   249                                           
   250      1000          927      0.9      0.0      if X is Y:  # shortcut in the common case euclidean_distances(X, X)
   251                                                   YY = XX.T
   252      1000          699      0.7      0.0      elif Y_norm_squared is not None:
   253      1000         8768      8.8      0.2          YY = np.atleast_2d(Y_norm_squared)
   254                                           
   255      1000         1956      2.0      0.1          if YY.shape != (1, Y.shape[0]):
   256                                                       raise ValueError(
   257                                                           "Incompatible dimensions for Y and Y_norm_squared")
   258                                               else:
   259                                                   YY = row_norms(Y, squared=True)[np.newaxis, :]
   260                                           
   261      1000      3174245   3174.2     83.0      distances = safe_sparse_dot(X, Y.T, dense_output=True)
   262      1000        93573     93.6      2.4      distances *= -2
   263      1000        69584     69.6      1.8      distances += XX
   264      1000        72021     72.0      1.9      distances += YY
   265      1000        92431     92.4      2.4      np.maximum(distances, 0, out=distances)
   266                                           
   267      1000         1575      1.6      0.0      if X is Y:
   268                                                   # Ensure that distances between vectors and themselves are set to 0.0.
   269                                                   # This may not be the case due to floating point rounding errors.
   270                                                   distances.flat[::distances.shape[0] + 1] = 0.0
   271                                           
   272      1000          804      0.8      0.0      return distances if squared else np.sqrt(distances, out=distances)
```

Disabling the profiler and using a coarser function timer we get the following timings:

Without Checks: 3.9655s
With Checks: 4.9669s

This is a 20% decrease in the amount of time taken (1 second total).

To ensure that this speedup was not just for parameters resembling my problem I
did a gridsearch on various parameter values and looked at the percent change.
For larget datasets the change is consistently positive. There are a few
negative changes for small datasets, but this is likely because of random
fluctuations. For datasets with at least a .1 second speed increase, there is a
15% average improvement with the improvement increasing for larger datasets.

```
    n_clusters  n_features  per_cluster  new_speed  old_speed  percent_change  absolute_change
0         2000         512          200  45.520926  56.996406        0.201337        11.475480
1         2000         512          100  46.009816  57.047427        0.193481        11.037611
2         2000         512           10  46.023527  56.965505        0.192081        10.941978
3         2000         512            1  46.036122  57.028242        0.192749        10.992120
4         2000         128          200  13.359815  16.861238        0.207661         3.501423
5         2000         128          100  13.594423  16.169258        0.159243         2.574835
6         2000         128           10  13.041987  16.357325        0.202682         3.315338
7         2000         128            1  13.517229  15.883567        0.148980         2.366338
8         2000          32          200   4.691806   5.384530        0.128651         0.692724
9         2000          32          100   4.785984   5.434597        0.119349         0.648613
10        2000          32           10   4.816490   5.309634        0.092877         0.493144
11        2000          32            1   4.678081   5.419021        0.136729         0.740940
12        2000           4          200   2.194186   2.311234        0.050643         0.117048
13        2000           4          100   2.195776   2.310168        0.049517         0.114392
14        2000           4           10   2.197284   2.313143        0.050087         0.115859
15        2000           4            1   2.245128   2.298406        0.023181         0.053278
16        1000         512          200  10.325005  13.263920        0.221572         2.938915
17        1000         512          100  10.372535  13.245843        0.216922         2.873308
18        1000         512           10  10.340922  13.195515        0.216331         2.854593
19        1000         512            1  10.377185  13.245806        0.216568         2.868621
20        1000         128          200   2.862865   3.559318        0.195670         0.696453
21        1000         128          100   2.814686   3.682527        0.235664         0.867841
22        1000         128           10   2.866434   3.692132        0.223637         0.825698
23        1000         128            1   3.000796   3.592812        0.164778         0.592016
24        1000          32          200   1.063762   1.172979        0.093111         0.109217
25        1000          32          100   1.053794   1.172330        0.101111         0.118536
26        1000          32           10   1.042551   1.164388        0.104636         0.121837
27        1000          32            1   1.045254   1.165895        0.103475         0.120641
28        1000           4          200   0.576731   0.612182        0.057909         0.035451
29        1000           4          100   0.576767   0.609157        0.053172         0.032390
30        1000           4           10   0.598036   0.607962        0.016327         0.009926
31        1000           4            1   0.578046   0.610441        0.053068         0.032395
32         100         512          200   0.124780   0.144742        0.137913         0.019962
33         100         512          100   0.122643   0.144813        0.153094         0.022170
34         100         512           10   0.123119   0.142057        0.133312         0.018938
35         100         512            1   0.122787   0.144781        0.151913         0.021994
36         100         128          200   0.049460   0.058768        0.158387         0.009308
37         100         128          100   0.060528   0.072691        0.167326         0.012163
38         100         128           10   0.049410   0.061536        0.197052         0.012126
39         100         128            1   0.052959   0.061433        0.137941         0.008474
40         100          32          200   0.029961   0.035051        0.145223         0.005090
41         100          32          100   0.028987   0.033165        0.125970         0.004178
42         100          32           10   0.030262   0.039366        0.231266         0.009104
43         100          32            1   0.028978   0.033200        0.127173         0.004222
44         100           4          200   0.022328   0.027136        0.177188         0.004808
45         100           4          100   0.015451   0.017483        0.116230         0.002032
46         100           4           10   0.014638   0.016714        0.124215         0.002076
47         100           4            1   0.014893   0.018615        0.199944         0.003722
48          10         512          200   0.003733   0.002868       -0.301604        -0.000865
49          10         512          100   0.002347   0.002843        0.174507         0.000496
50          10         512           10   0.002344   0.002907        0.193718         0.000563
51          10         512            1   0.002358   0.006877        0.657086         0.004519
52          10         128          200   0.001717   0.002016        0.148196         0.000299
53          10         128          100   0.007036   0.002073       -2.394020        -0.004963
54          10         128           10   0.001862   0.002053        0.093021         0.000191
55          10         128            1   0.001747   0.002061        0.152244         0.000314
56          10          32          200   0.001596   0.001831        0.128385         0.000235
57          10          32          100   0.001543   0.001787        0.136606         0.000244
58          10          32           10   0.001550   0.001786        0.132159         0.000236
59          10          32            1   0.001551   0.001857        0.164719         0.000306
60          10           4          200   0.001525   0.001715        0.110663         0.000190
61          10           4          100   0.007363   0.001704       -3.320974        -0.005659
62          10           4           10   0.001505   0.001696        0.112470         0.000191
63          10           4            1   0.001494   0.009352        0.840255         0.007858
```

The script I used to generate these numbers is:

``` python
"""
python speedup_kmeans.py --profile
python speedup_kmeans.py

For 1000 clusters and 10000 datapoints:
    5 seconds with checks
    3 seconds without checks
    3 seconds without checks and with out safe_dot
"""
from __future__ import absolute_import, division, print_function, unicode_literals
import utool as ut
import sklearn  # NOQA
from sklearn.datasets.samples_generator import make_blobs
from sklearn.utils.extmath import row_norms, squared_norm  # NOQA
import sklearn.cluster
import numpy as np
from sklearn.metrics.pairwise import euclidean_distances  # NOQA
(print, rrr, profile) = ut.inject2(__name__, '[tester]')


#@profile
def test_kmeans_plus_plus_speed(n_clusters=1000, n_features=128, per_cluster=10, fix=True):
    # Make random cluster centers on a ball
    rng = np.random.RandomState(42)
    centers = rng.rand(n_clusters, n_features)
    centers /= np.linalg.norm(centers, axis=0)[None, :]
    (centers * 512).astype(np.int) / 512
    centers /= np.linalg.norm(centers, axis=0)[None, :]

    n_samples = n_clusters * 10
    n_clusters, n_features = centers.shape
    X, true_labels = make_blobs(n_samples=n_samples, centers=centers,
                                cluster_std=1., random_state=42)

    x_squared_norms = row_norms(X, squared=True)

    _k_init = sklearn.cluster.k_means_._k_init
    random_state = rng
    n_local_trials = None  # NOQA

    #print('Testing kmeans init')
    with ut.Timer('testing kmeans init') as t:
        centers = _k_init(X, n_clusters, random_state=random_state, x_squared_norms=x_squared_norms, fix=fix)
    #print('Done testing kmeans init')
    return t.ellapsed


def main():
    basis = {
        'n_clusters': [10, 100, 1000, 2000][::-1],
        'n_features': [4, 32, 128, 512][::-1],
        'per_cluster': [1, 10, 100, 200][::-1],
    }
    vals = []
    for kw in ut.ProgIter(ut.all_dict_combinations(basis), lbl='gridsearch',
                          bs=False, adjust=False, freq=1):
        print(kw)
        new_speed = test_kmeans_plus_plus_speed(fix=True, **kw)
        old_speed = test_kmeans_plus_plus_speed(fix=False, **kw)
        kw['new_speed'] = new_speed
        kw['old_speed'] = old_speed
        vals.append(kw)

    import pandas as pd
    pd.options.display.max_rows = 64
    pd.options.display.width = 100
    pd.options.display.max_ = 64
    df = pd.DataFrame.from_dict(vals)
    df['percent_change'] = (df['old_speed'] - df['new_speed']) / df['old_speed']
    df = df.reindex_axis(['n_clusters', 'n_features', 'per_cluster', 'new_speed', 'old_speed', 'percent_change'], axis=1)
    df['absolute_change'] = (df['old_speed'] - df['new_speed'])
    print(df)

    print(df['percent_change'][df['absolute_change'] > .1].mean())
    #print(df.loc[df['percent_change'].argsort()[::-1]])

    #try:
    #    profile.dump_stats('out.lprof')
    #    profile.print_stats(stripzeros=True)
    #except Exception:
    #    pass

#if __name__ == '__main__':
#    main()
```

<!-- Reviewable:start -->

---

This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/scikit-learn/scikit-learn/7383)

<!-- Reviewable:end -->

user: Erotemic
issue_7382: state: open
title: Add prefit to VotingClassifier
body: EDIT:  Never mind, saw that I need to call fit on it and not just pass in already fitted models, which is how I wanted to used it.  Apologies.  

Looking at the voting_classifier.py

class VotingClassifier(BaseEstimator, ClassifierMixin, TransformerMixin):
    """Soft Voting/Majority Rule classifier for unfitted estimators.

```
.. versionadded:: 0.17

Read more in the :ref:`User Guide <voting_classifier>`.

Parameters
----------
estimators : list of (string, estimator) tuples
    Invoking the ``fit`` method on the ``VotingClassifier`` will fit clones
    of those original estimators that will be stored in the class attribute
    `self.estimators_`.
```

....

```
def __init__(self, estimators, voting='hard', weights=None):

    **self.estimators = estimators**
    self.named_estimators = dict(estimators)
    self.voting = voting
    self.weights = weights

def _predict(self, X):
    """Collect results from clf.predict calls. """
    return np.asarray([clf.predict(X) for clf in self.estimators_]).T
```

So it looks like the initialization of self.estimators should be self.estimators_

My code:

import numpy
import sys

from collections import OrderedDict
from scipy import special, stats
from scipy.stats import f_oneway
from sklearn.utils import safe_mask
from sklearn.feature_selection import SelectKBest, f_classif, f_regression
from sklearn import svm
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix
from sklearn.grid_search import GridSearchCV
import sklearn.svm
from sklearn.feature_selection import RFE

from sklearn.cross_validation import StratifiedShuffleSplit
from sklearn.ensemble import VotingClassifier

from scipy.stats.stats import pearsonr

from DataStream import *
from Config import genders

def training2(X, y, test_X = None, test_y = None):

```
sss = StratifiedShuffleSplit(train_y, 3, test_size=0.5, random_state=0)
estimators = []
i = 0
for train_index, test_index in sss:
    i += 1
    print train_index
    print(i)

    clf = svm.NuSVC() #gamma=0.5, nu=0.8)
    clf.fit(X.astype(float), y.astype(float))
    #print(clf)
    estimators.append(('clf'+str(i), clf))

eclf = VotingClassifier(estimators=estimators, voting='hard')
print(estimators)


if test_X is not None and test_y is not None:
    print(testing(X=test_X, y=test_y, clf=eclf))

return clf
```
#### Actual Results

Traceback (most recent call last):
  File "/u/arvie/PHD/DCCA_Experiment2/expA_SVM.py", line 165, in <module>
    my_svm = training2(apply_feature_selection(train_X, features), train_y, apply_feature_selection(test_X, features), test_y)
  File "/u/arvie/PHD/DCCA_Experiment2/expA_SVM.py", line 57, in training2
    print(testing(X=test_X, y=test_y, clf=eclf))
  File "/u/arvie/PHD/DCCA_Experiment2/expA_SVM.py", line 116, in testing
    return accuracy_score(y, clf.predict(X), normalize=True)
  File "/u/arvie/.local/lib/python2.7/site-packages/sklearn/ensemble/voting_classifier.py", line 149, in predict
    predictions = self._predict(X)
  File "/u/arvie/.local/lib/python2.7/site-packages/sklearn/ensemble/voting_classifier.py", line 226, in _predict
    return np.asarray([clf.predict(X) for clf in self.estimators_]).T
AttributeError: 'VotingClassifier' object has no attribute 'estimators_'
#### Versions

import sys; print("Python", sys.version)
('Python', '2.7.6 (default, Jun 22 2015, 17:58:13) \n[GCC 4.8.2]')

> > > import numpy; print("NumPy", numpy.**version**)

('NumPy', '1.11.0')

> > > import scipy; print("SciPy", scipy.**version**)
> > > ('SciPy', '0.17.0')
> > > import sklearn; print("Scikit-Learn", sklearn.**version**)
> > > ('Scikit-Learn', '0.17.1')

user: arvieFrydenlund
issue_7366: state: open
title: Predictions from RandomForestClassifier completely unstable between different machines
body: **Text classification - Entity Recognition**

I am currently having some text classification experiments using windows of word embeddings + some hand-crafted features. I already used LinearSVC and LogisticRegression and I want to finalize my experiments with RandomForestClassifier.

**Use of VM - Dataset (train,validation,test) - Weird behaviour**

Because my datasets are pretty big around 15-35 GB, I am having my training online on a VM and then I make a classification report on my test set on my laptop.

So far so good with LinearSVC and LogisticRegression, but RandomForestClassifier has pretty weird behaviour.

After training classifier in my code, I make a classification report over a validation set (the last 20% of my initial samples). Of course I don't use the same samples during the training, so we have 20% of fresh samples to measure our classifier.  WIth RandomForestClassifier I get the following report:

**VALIDATION RESULTS**

```
     precision    recall  f1-score   support

  0       0.99      0.99      0.99     78473
  1       0.89      0.81      0.85      5711
```

Then I downloaded the classifier, loaded and start the classification report over test set on my laptop, which gives us:

**TEST RESULTS (LAPTOP)**

```
     precision    recall  f1-score   support

  0       0.93      0.99      0.96     33845
  1       0.52      0.07      0.13      2618
```

This doesn't look good at all. There is no any possibility to have so much different results between validation and test  from the same dataset (both unseen during training). So I tried to think what possibly is going wrong from a theoretical point of view and I couldn't find something that makes such a great difference. So I tried to run the same test on the VM which I use for training experiments.

**TEST RESULTS (VM)**

```
     precision    recall  f1-score   support

  0       0.99      0.99      0.99     33845
  1       0.91      0.81      0.85      2618
```

So it seems (in my case?) RandomForestClassifier works really differently from one machine to the other. **Any other classifier (SVC, LR) gives the same validation report on both machines.**

I would be grateful to have an explanation of this strange behavior to fix this issue. Maybe I'm missing something...

**Extra Information**
- Enviroments

VM : 
OS: Ubuntu 14.04 LTS 
Python 3.5
scikit-learn  0.17.1
numpy 1.11.1
scipy 0.17.1

LAPTOP : 
OS: Ubuntu 16.04 LTS
Python 3.5
scikit-learn  0.17.1
numpy 1.11.1
scipy 0.17.1
- Parameters

RandomForestClassifier(n_jobs=1, n_estimators=200, warm_start= True)

_I was using n_jobs=-1, but then I fixed it to one to be sure that the problem is not happening because of the multi-thread difference (4 threads vs 8 threads)._
- Save/Load

I'm saving - loading classifiers as suggested on scikit-learn.org ( [http://scikit-learn.org/stable/modules/model_persistence.html](url) )
- Metrics

Default scikit-learn classification_report (metrics.classification_report)
- Notice (Size of .npy files)

LinearSVC and LogisticRegression are saved in 1 .pkl file and 4 extra .npy files for each classifier. RandomForestClassifier is saved 1 .pkl file and 801(!!!) extra .npy files.

user: iliaschalkidis
issue_7343: state: open
title: Add precision at K into scikit-learn
body: We currently have [average_precision_score](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html) in our scikit-learn metrics, but it doesn't seem possible to calculate average precision at k.
# Example

```
>>> from precision_at_k import average_precision_at_k
>>> y_true = [[0, 0, 1, 1], \
              [0, 1, 0, 0]]
>>> y_pred = [[0.5, 0.0, 1.0, 0.0], \
              [1., 1.1, 0.5,   0.0]]
>>> average_precision_at_k(y_true, y_pred, 2)
0.75
```

This value is found by taking the precision of each set of true labels / predicted label pairs:
- `precision_at_k([0, 0, 1, 1], [0.5, 0.0, 1.0, 0.0], k=2) = 0.5` 
- `precision_at_k([1, 1, 0, 0], [0.0, 1.1, 1.0, 0.0], k=2) = 1`

and then taking the average of these values (which equals `0.75`).

When there is more than 1 true label, there are several corner cases to handle.
# References
- [WSABIE: Scaling up to large scale vocabulary image annotation](http://www.thespermwhale.com/jaseweston/papers/wsabie-ijcai.pdf) (This paper assumes that there is only one true label value, but my example above assumes that there may be multiple.)

Thoughts? Alternatively, if this function is already in the library, and I have missed it, I would love to hear about it.
## Edits
- Changed the example to reflect predicted confidence scores rather than binary predicted scores.
- Changed the title of this post: "Add average precision at K into scikit-learn" -> "Add precision at K into scikit-learn" 

user: hlin117
issue_7337: state: open
title: Fix feature importances computation error in RandomForest, ExtraTrees…
body: … and GradientBoosting

This fixes an error in feature importances computation of forests (i.e. RandomForest, ExtraTrees and GradientBoosting). As it is now, the feature importances are computed as the mean over the trees of the _normalized_ feature importances of each tree.

This is wrong especially for GradientBoosting as it means that the feature importances of the lastest trees will be weighted the same as the ones of the first trees although the lastest splits contribute much less to the global reduction of impurity. This is why the normalization must be done at the end.
(although it is not exactly the same computation, this is what is done in J. Friedman, “Greedy Function Approximation: A Gradient Boosting Machine”, The Annals of Statistics, Vol. 29, No. 5, 2001. in equation 44 and 45)

Thus, I have added a method to `BaseDecisionTree` called `_compute_feature_importances` that does the same as the attribute `feature_importances_` except that it accepts a facultative `normalize` argument (`feature_importances_` has been kept). `BaseForest` and `BaseGradientBoosting` now use that method to compute the non-normalized feature importances of each tree, sum them and normalize them before returning them.

user: tlorieul
labels: Needs Decision: Requires decision
issue_7308: state: open
title: RFE/RFECV doesn't work with sample weights
body: As far as I can tell, `sklearn.feature_selection.RFE` has no way to pass sample weights to the estimator alongside the data.

I have fixed this in my code with:

``` diff
index bbe0cda..f5072b2 100644
--- a/sklearn/feature_selection/rfe.py
+++ b/sklearn/feature_selection/rfe.py
@@ -120,7 +120,7 @@ class RFE(BaseEstimator, MetaEstimatorMixin, SelectorMixin):
     def _estimator_type(self):
         return self.estimator._estimator_type

-    def fit(self, X, y):
+    def fit(self, X, y, **fit_params):
         """Fit the RFE model and then the underlying estimator on the selected
            features.

@@ -132,9 +132,9 @@ class RFE(BaseEstimator, MetaEstimatorMixin, SelectorMixin):
         y : array-like, shape = [n_samples]
             The target values.
         """
-        return self._fit(X, y)
+        return self._fit(X, y, **fit_params)

-    def _fit(self, X, y, step_score=None):
+    def _fit(self, X, y, step_score=None, **fit_params):
         X, y = check_X_y(X, y, "csc")
         # Initialization
         n_features = X.shape[1]
@@ -166,7 +166,7 @@ class RFE(BaseEstimator, MetaEstimatorMixin, SelectorMixin):
             if self.verbose > 0:
                 print("Fitting estimator with %d features." % np.sum(support_))

-            estimator.fit(X[:, features], y)
+            estimator.fit(X[:, features], y, **fit_params)

             # Get coefs
             if hasattr(estimator, 'coef_'):
```

Would this be a worthwhile contribution to scikit-learn?
#### Versions

<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->

``` python

In [1]: import platform; print(platform.platform())
Linux-3.13.0-63-generic-x86_64-with-Ubuntu-14.04-trusty

In [2]: import sys; print("Python", sys.version)
('Python', '2.7.6 (default, Jun 22 2015, 17:58:13) \n[GCC 4.8.2]')

In [3]: import numpy; print("NumPy", numpy.__version__)
('NumPy', '1.11.0')

In [4]: import scipy; print("SciPy", scipy.__version__)
('SciPy', '0.17.1')

In [5]: import sklearn; print("Scikit-Learn", sklearn.__version__)
('Scikit-Learn', '0.18.dev0')
```

<!-- Thanks for contributing! -->

TODO:

- [x] Add support for `sample_weight` in `RFE`
- [ ] Add support for `sample_weight` in `RFECV`

user: g-rutter
issue_7284: state: open
title: [DOC] GSoC BayesianGaussianMixture formula
body: This PR adds the doc concerning the derivation formula of the `BayesianGaussianMixture` class.

I've done a separate PR because this is just formula and I thought it wil be easier to review if it was extracted on another PR.

Of course, this can't be merge before #6651.

user: tguillemot
issue_7206: state: open
title: `results_` attribute for all EstimatorCV classes?
body: In light of #6697 
1. ~~Should we have a `BaseEstimatorCV` class defining the required parameters (like `results_`, `best_estimators_`) as an interface attribute that can be overwritten?~~
2. Should the parameter still be called `results_` and not `search_results_` - Let's call it `cv_results_` would be more representative of what it holds.

**TODO**
- [ ] calibration.py:class `CalibratedClassifierCV(BaseEstimator, ClassifierMixin)`
- [ ] covariance/graph_lasso_.py:class GraphLassoCV(GraphLasso):
- [ ] feature_selection/rfe.py:class RFECV(RFE, MetaEstimatorMixin):
- [ ] linear_model/coordinate_descent.py:class LassoCV(LinearModelCV, RegressorMixin):
- [ ] linear_model/coordinate_descent.py:class ElasticNetCV(LinearModelCV, RegressorMixin):
- [ ] linear_model/coordinate_descent.py:class MultiTaskElasticNetCV(LinearModelCV, RegressorMixin):
- [ ] linear_model/coordinate_descent.py:class MultiTaskLassoCV(LinearModelCV, RegressorMixin):
- [ ] linear_model/least_angle.py:class LarsCV(Lars):
- [ ] linear_model/least_angle.py:class LassoLarsCV(LarsCV):
- [ ] linear_model/logistic.py:class LogisticRegressionCV(LogisticRegression, BaseEstimator,
- [ ] linear_model/omp.py:class OrthogonalMatchingPursuitCV(LinearModel, RegressorMixin):
- [ ] linear_model/ridge.py:class RidgeCV(_BaseRidgeCV, RegressorMixin):
- [ ] linear_model/ridge.py:class RidgeClassifierCV(LinearClassifierMixin, _BaseRidgeCV):

@amueller @jnothman @MechCoder @vene @GaelVaroquaux @agramfort 

user: raghavrv
issue_7188: state: open
title: Adding Correlation Coefficient (CC) as a metric for regression
body: The Correlation Coefficient (CC) although not a particularly good metric due to it being scale and offset invariant is sometimes used as a goodness-of-fit metric in regression tasks. 

Perhaps a feature worth adding? Implementation should be straightforward since it would essentially be a mere call to numpy's `corrcoeff` [function](http://docs.scipy.org/doc/numpy/reference/generated/numpy.corrcoef.html), followed by appropriate indexing (i.e. `[0,1]`). 

user: agamemnonc
issue_7178: state: open
title: [MRG] ENH support multiclass column vector array-like in metrics
body: For non-thresholded classification metrics, this adds tests for:
- column vectors as lists of lists
- multiclass input

and ensures their support. This effectively fixes #7081.

Column vectors are analogous to multioutput classification targets with a single target.

We did not previously support nested lists (unlike arrays) as a target format because these were confusable with the now-unsupported-for-some-time list of lists multilabel format. I'm not _entirely_ sure they should be supported now, but it seems consistent with our support of "array-likes" elsewhere.

user: jnothman
issue_7153: state: open
title: [WIP] Make neighbor tree module supports fused types
body: Ball Tree and KD Tree currently only support `np.float64`, which is a waste of memory as same quality results should be obtainable with `np.float32`.

**Note**: `self.new_data` and `self.new_data_arr` are just for development usage, I try to first use them to repeat what `self.data` and `self.data_arr` do. With this manner, I can still build scikit-learn successfully and I'll change `self.data` and `self.data_arr` in the end.

user: yenchenlin
issue_7139: state: open
title: add common test: fixing random_state makes algorithms deterministic.
body: I think we should add a test to see if all estimators either are deterministic and have no random_state, or are deterministic after setting the random_state.

user: amueller
labels: Needs Decision: Requires decision
Hard: Hard level of difficulty
issue_7136: state: open
title: `fit_params` in conjunction with `FeatureUnion`
body: #### Description

Using `fit_params` in conjunction with `FeatureUnion` may not work as expected. It would be helpful if `fit_params` names would be resolved to fit the estimators in `transformer_list`.
#### Steps/Code to Reproduce

```
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.preprocessing import FunctionTransformer
from sklearn.datasets import make_classification

X, y = make_classification()

class MyTransformer(FunctionTransformer):
    def fit(self, X, y=None, **fit_params):
        print("Fit params are: ", fit_params)
        return super().fit(X, y)

pipe = Pipeline([
    ('step0', FunctionTransformer()),
    ('step1', FeatureUnion([
        ('feature0', MyTransformer()),
        ('feature1', MyTransformer()),
    ])),
    ('step2', FunctionTransformer()),
])

pipe.fit(X, y, step1__feature1__someparam=123)
```
#### Expected Results

```
# prints
Fit params are:  {'someparam': 123}
```
#### Actual Results

```
# prints
Fit params are:  {'feature1__someparam': 123}  # by feature0
Fit params are:  {'feature1__someparam': 123}  # by feature1
```
#### Comment

Maybe the actual outcome is what it is supposed to be, but my expectation would be that `FeatureUnion` resolves the estimator name and only passes the `fit_params` to the corresponding estimator. At least it would be very useful if it did.
#### Versions

Python 3.4.5
NumPy 1.10.4
SciPy 0.17.0
Scikit-Learn 0.17.1

user: BenjaminBossan
issue_7116: state: open
title: Discussion of useful plotting
body: In #6338 we discussed whether we want plotting code in sklearn.
We decided that the answer is yes, and we add a plotting submodule.
I think it would be good to discuss what functions to add.

Here are some that come to mind:
- better 2d scatterplot (with legend) - I have that for my book
- confusion matrix (have that too)
- plotting coefficients from a linear model or feature importances (maybe? It's relatively simple)
- plotting results from grid-search
- plotting learning curves
- plotting 2d decision functions
- plotting trees in the notebook (currently getting them into a notebook is a bit of a pain, though the notebook actually can plot trees directly)

Not sure if there are other common types of plots. Any input welcome.

Also: we have quantile plots that we might want to move?

user: amueller
issue_7107: state: open
title: Refactored Count Vectorizer to be more memory efficient on N-grams
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue

<!-- Example: Fixes #1234 -->
#### What does this implement/fix? Explain your changes.
#### Any other comments?

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

When max_features are specified, CountVectorizer will first count the word occurrence. It will take the the top occurring words up to max_features.  It will only make a 2-gram when its 1 gram count is apart of max_features, make a 3-gram when its 2-gram is apart of the max features and so on. This faster and more memory efficient on n-grams when the data sets are large.

user: qingyili
issue_7087: state: open
title: Biclustering n_components default
body: The `n_components` parameter of the biclustering classes is 3 by default. I think it should be `n_clusters` by default. @kemaleren wdyt?

user: amueller
issue_7081: state: open
title: Different Precision and Recall values for the same dataset
body: #### Description

Varying outputs for the same train and test data
#### Steps/Code to Reproduce

Example:

```
import pandas as pd
import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.classification import precision_recall_fscore_support
from sklearn import metrics

newsgroups_train = fetch_20newsgroups(subset='train')
newsgroups_test = fetch_20newsgroups(subset='test')
X_train = newsgroups_train.data
X_test = newsgroups_test.data
y_train = list(newsgroups_train.target)
y_test = list(newsgroups_test.target)
y_train = [[x] for x in y_train]
y_test = [[x] for x in y_test]

vectorizer = TfidfVectorizer(ngram_range = (1,1))
X_train = vectorizer.fit_transform(X_train)
X_test = vectorizer.transform(X_test)
clf = OneVsRestClassifier(LogisticRegression(C=(6), penalty='l1')).fit(X_train, y_train)
pred = clf.predict(X_test)
overall_precision,overall_recall,overall_fscore,overall_support = precision_recall_fscore_support(y_test,pred,average='weighted')
print "Precision " + str(overall_precision) + " " + "Recall " + str(overall_recall)
```
#### Expected Results

Precision 0.809072982209 and Recall 0.80669144981
#### Actual Results

Precision 0.881659860014 and Recall 0.631439192777
Passing y_test and y_train as just a list gives the Expected Results, whereas converting the y_train and y_test as a list of lists gives different results. I think it is because it understands that one is Multilabel and other is not, but shouldn't the results be the same?
#### Versions

Darwin-15.5.0-x86_64-i386-64bit
Python 2.7.1
'NumPy', '1.11.0'
'SciPy', '0.17.1'
'Scikit-Learn', '0.16.1'

<!--
Please run the following snippet and paste the output below.


-->

<!-- Thanks for contributing! -->

user: HimaVarsha94
issue_7078: state: open
title: [MRG] addressing issue #6887, adding callable support for covariance comp.
body: #### Reference Issue
#6887
#### What does this implement/fix? Explain your changes.

Adds support for a callable computing covariance matrix.
#### Any other comments?

user: btabibian
issue_7059: state: open
title: Any way to make BinaryTree support fused type?
body: Hello guys, currently I'm trying to make neighbors tree algorithms to support Cython fused types so that the memory needed can be drastically reduced.

However, both `KDTree` and `BallTree` are subclasses of `BinaryTree`, and [this line](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/binary_tree.pxi#L1031) of `BinaryTree` seems to decide the datatype (`np.float64`) of data it stores at initialization time, i.e., even before it sees the actual datatype of training data (which may be `np.float32`).

The reason to initialize array in `__cinit__` is explained in the comments of code:

> ```
>   # Use cinit to initialize all arrays to empty: this will prevent memory
>      # errors and seg-faults in rare cases where __init__ is not called
> ```

I wonder what is the `rare cases where __init__ is not called`?
Can we handle those rare cases as special cases and move the data initialization to `__init__`, so that it can init a array of datatype of training data?

Thanks in advance.
ping @jakevdp @jnothman @MechCoder 

user: yenchenlin
labels: float32: Issues related to support for 32bit data
issue_7058: state: open
title: will largevis be implemented in the near future?
body: https://arxiv.org/abs/1602.00370
![image](https://cloud.githubusercontent.com/assets/2433806/17012965/eccc495a-4f4d-11e6-8dd0-025c7c6e63f6.png)
seems like _largevis_ is competitive with _t-sne_.

And there's a cpp implementation on github [lferry007/LargeVis](https://github.com/lferry007/LargeVis)

is there any plan on implementing _largevis_ in sklearn

user: wlbksy
issue_7052: state: open
title: Link Gaussian Naive Bayes to fMRI literature
body: We don't have a good reference for GaussianNB iirc. We should check the references in the fMRI literature and see if they use some smoothing or anything like that.

user: amueller
issue_7050: state: open
title: PLSRegression VIP score calculation
body: I was looking for an implementation of VIP (Variable Importance in the Projection) scoring for PLS models as described in [this publication](http://www.sciencedirect.com/science/article/pii/S0169743912001542). 

The algorithm from that paper is implemented in [this MATLAB code](https://www.mathworks.com/matlabcentral/fileexchange/47767-libpls-1-95-zip/content/libPLS_1.95/vip.m) which is in The MATLAB code is available under a [BSD License](https://www.mathworks.com/matlabcentral/fileexchange/view_license?file_info_id=47767). Below is a Python implementation of this same code, which produces the VIP scores from scikit-learn PLSRegression models.

```
def vip(x, y, model):
    t = model.x_scores_
    w = model.x_weights_
    q = model.y_loadings_

    m, p = x.shape
    _, h = t.shape

    vips = np.zeros((p,))

    s = np.diag(t.T @ t @ q.T @ q).reshape(h, -1)
    total_s = np.sum(s)

    for i in range(p):
        weight = np.array([ (w[i,j] / np.linalg.norm(w[:,j]))**2 for j in range(h) ])
        vips[i] = np.sqrt(p*(s.T @ weight)/total_s)

    return vips
```

Firstly, is this something that would be considered for addition to scikit-learn, and if so, where would be a good place to add it? There was a [discussion on the mailing list from 2015](https://sourceforge.net/p/scikit-learn/mailman/message/33752606/) about adding this, but it doesn't look like it got any further.

user: mfitzp
issue_6992: state: open
title: ENH Isomap: adding choice option for neighborhood function (plus some code cleaning)
body: This pull request proposes 3 changes:
1. In the same way as the [original Tenenbaum's implementation and paper of isomap](http://isomap.stanford.edu/), I added the option to choose the type of neighborhood graph (kneighbor or radius), which can be really useful.
2. Removing the redundant use of NearestNeighbors object and function kneighbors_graph, the latter already existing as a method in NearestNeighbors
3. Structuring+commenting the code to easily get the main steps of isomap

user: cvanwynsberghe
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_6986: state: open
title: several issues with Dictionary Learning implementation
body: Regarding the sklearn.decomposition.dict_learning.py module:
1. currently, both the DictionaryLearning and MiniBatchDictionaryLearning classes only support LARS and Coordinate Descent algorithms for Sparse Coding when learning the dictionary (through the fit_algorithm argument). However, the Online Dictionary Learning algorithm of Mairal et al, can also be used with zero-norm sparse coders (e.g OMP) that are usually faster.
2. we should add another dictionary initialization procedure (the only option is the SVD of the data matrix) where we use a subset of the datapoints as the initial atoms of the dictionary. From experience, this initialization works better than SVD, at least with visual data. 

So what do you think? Should I start implementing this?

user: ektormak
issue_6972: state: open
title: HashingVectorizer uses l2 norm, Countvectorizer doesn't. Thats confusing
body: Maybe we should add normalize to CountVectorizer? I'm not sure. I think it is very counterintuitive that HashingVectorizer is not a plug-in replacement for CountVectorizer.

user: amueller
issue_6948: state: open
title: [WIP] ENH Optimal n_clusters values
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue

<!-- Example: Fixes #1234 -->

I propose to integrate the algorithms which compute automatically the `n_clusters` values proposed in #4301 by @afouchet.

This PR is the the second step of the work initiated by #6823.
#### What does this implement/fix? Explain your changes.

I propose several algorithms that analyses the best value for 'n_clusters' :
- max value for some unsupervised metrics (silhouette, calinski and harabraz)
- stability criterium (using folwkes and mallows metric) 
- max value of the distortion jump
- gap criterium
- pham criterium
#### Any other comments?

TODO:
- [X] Use parallel to accelerate the computation
- [X] Not range parameters for `n_clusters`
- [X] Using ParameterGrid class allowing to see the evolution of other parameters
- [X] Adapted to have a similar use than GridSearch
- [x] The results can be analysed by pandas
- [ ] docstring
- [ ] docs
- [ ] tests
- [ ] examples

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: tguillemot
issue_6930: state: open
title: [MRG] Use logging.info instead of print (#6929)
body: <!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue

<!-- Example: Fixes #1234 -->

Fixes #6929
#### What does this implement/fix? Explain your changes.

Replaces 'print' with 'logging.info' so the library will no longer print to stdout.
#### Any other comments?

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

user: SahilKang
labels: Needs Decision: Requires decision
issue_6929: state: open
title: Library should not be printing to stdout without verbose option
body: I think it's good policy not to print anything to standard output in library code, except when that's specified by the API, e.g. through a `verbose` parameter. We have, for instance, a few unguarded `print` calls in `sklearn/datasets`. These should be weeded out. One option for fixing is to use `logging.info`; another is to use `print(..., file=sys.stderr)`.

user: jnothman
issue_6915: state: open
title: Automatically set 'V' parameter to cov(X) when using mahalanobis distance metric
body: Would it be considered bad design to automatically set the 'V' parameter for the mahalanobis distance metric to cov(X) for all estimators that (possibly) use this metric?

I'm asking because it would facilitate the use of the mahalanobis distance in cross validation scenarios in general (because one won't have to set the V parameter over and and over again for each fold), or more specifically when used with GridSearchCV (because with the way things are now, it won't work with a KNeighborsClassifier using the malahanobis distance, for example). 

On the other hand, adding code to many estimators' `fit()` methods just to fix this minor issue seems a bit messy to me. Could there be a more elegant way? I'd be glad to take suggestions and follow them up with a PR.

user: jln-ho
issue_6891: state: open
title: Stratification in Nystroem kernel approximation
body: Would it be possible to add stratification in Nystroem kernel approximation? The `fit` method already takes `y` as an argument but doesn't do anything with it. Perhaps instead it could be used so that if `y` is supplied, the sampling in `X` is guided such that the data points reflect an even number of samples per class.

user: thomasbkahn
issue_6887: state: open
title: GraphLasso with robust correlation estimates
body: This was on the mailing list and seemed like it shouldn't go in the trash:

```
Hi all,

I'm using GraphLASSO to estimate the graphical model and precision 
matrix of my variables. It is well known that GraphLASSO and related 
methods are very sensitive to contaminated data and their estimates have 
low break-down points:
http://arxiv.org/abs/1501.01219

As suggested by the authors in the above paper I'd like to use 
GraphLASSO with robust correlation matrices to estimate the precision 
matrix.  When I looked at the source code of GraphLASSO however, it 
seemed like there's no way to supply the fit method with a 
pre-calculated correlation matrix, as it is calculated internally and 
automatically using the empirical_covariance() method.

I know that if I rank my data column-wise before applying GraphLASSO 
I'll get Spearman correlation. I tried this, and it already improves the 
performance of GraphLASSO with outliers, so I'd advise adding this trick 
to the documentation to raise awareness of this issue.

But the authors suggest the Gaussian rank correlation works even better 
than Spearman, because it handles normally distributed variables better, 
while still having high break-down point as Spearman. Therefore I'd like 
to calculate a Gaussian rank correlation matrix and supply it to the 
GraphLASSO method. Is there any way with the current implementation to 
do this or should I rewrite the GraphLASSO class to make this possible?

Cheers,
Daniel

```

user: amueller
issue_6868: state: open
title: GaussianProcesss: active_dims?
body: When doing kernel engineering, you often want different parts of the kernel to operate on different features (like some on time and some on space and some on whatever).
From what I can see, this is not currently possible with sklearn.
GPy implements that with a `active_dims` option for each kernel.
That seems relatively easy, and I think it would be pretty cool. ping @jmetzen (also @jnothman and @GaelVaroquaux for the voice of reason?)

user: amueller
issue_6845: state: open
title: Orthogonal projection to latent structures discriminant analysis.
body: Are there any plans to implement O-PLS in Sklearn?

Thank you,

Keiron.

user: KeironO
issue_6832: state: open
title: [MRG] Documentation and some input validation for get_scorer
body: # Roadmap
- [x] Added documentation for `sklearn.metrics.get_scorer`
- [ ] Input validation check
  - [x] `scoring` parameter should be a callable
  - [ ] `scoring` should only receive 3 parameters
  - [ ] Add test cases for input validation
## Original Post

Added documentation for `sklearn.metrics.get_scorer`, which (until now) seemed blank. (Link [here](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.get_scorer.html#sklearn.metrics.get_scorer).)

To be honest, I can't seem to find a user case where this function would be useful. For example, if we were to use `cross_val_score`, we can state `cross_val_score(..., scoring="accuracy")` instead of `cross_val_score(..., scoring=get_scorer("accuracy")` to evaluate using the accuracy score. On the other hand, if a user generated his or her own evaluation metric, it would make more sense to wrap it with `make_scorer`. 

@GaelVaroquaux, do you have any ideas? (My idea is to deprecate the function altogether.)

If we choose to move forward with documenting this function, we probably want to make sure that if `scoring` is not a string, that it is a callable that takes in three arguments.

Motivation for PR: #6697.

user: hlin117
issue_6830: state: open
title: make LinearSVC(dual="auto") default
body: I think the `dual` variable in LinearSVC and LogisticRegression should be set based on the `penalty` and `loss` parameters. (not sure if there is already an issue for that).

user: amueller
issue_6828: state: open
title: Weighting the Smacof MDS algorithm
body: Hi!

My name is Charles, and I would like to share my very first contribution to open-source project.
#### Smacof from sklearn rules ! It only lacks the weighting

I regularly need the use of SMACOF algorithm, for sensor localization. In this field (and others), the weighting is mandatory to give more or less importance to pairwise distances. I send you to a well-known paper about it : [Costa2006](https://scholar.google.fr/scholar?q=distributed+weighted-multidimensional+scaling+for+node+localization+in+sensor+networks&btnG=&hl=en&as_sdt=0%2C5)

So far I have been using a wrapping of smacof in R with rpy2, because it features the weighting. But everytime I update R or rpy2 I get errors, about types, etc.

You added a very good implementation of smacof in the pull request #797. The idea of adding the weighting was proposed, but was not done, was there maybe a reason about it ?
#### How to feature it

Today I tried to modify the source code of sklearn, and my tests on sensor localization, by adding weighting have shown consistent results so far.
The idea could be to add a weight parameter at the fit method: 

``` python
from sklearn.manifold import MDS

smacof = MDS(n_components=2, dissimilarity='precomputed')

# Let D be a n_sample x n_sample euclidean distance matrix
# and weighting be a n_sample x n_sample weight matrix
smacof.fit(D, weight=weight) 
pl.scatter(*smacof.embedding_.T) # maps the sensors
```
#### What to change?

Not that much actually! The way it was coded only need few modifications in the _smacof_single() function.

As you can see in the [algorithm description](https://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&ved=0ahUKEwiW3fX07vfMAhUCJ5oKHRRJBqEQFghCMAQ&url=http%3A%2F%2Fstatistics.ucla.edu%2Fsystem%2Fresources%2FBAhbBlsHOgZmSSJYMjAxMi8wNS8xNi8xNV8xNl8yOF8yMDdfTXVsdGlkaW1lbnNpb25hbF9TY2FsaW5nX1VzaW5nX01ham9yaXphdGlvbl9TTUFDT0ZfaW5fUi5wZGYGOgZFVA%2FMultidimensional%2520Scaling%2520Using%2520Majorization%3A%2520SMACOF%2520in%2520R.pdf&usg=AFQjCNHAKeBatnwzXQqrRefrWGwdD5KqAA), it needs to compute the matrix V and its pseudo inverse, for the final update (eq. (14)). And also, it needs to modify B, by weighting the `ratio` variable (=s_ij eq. (11) ).

Are you interested in adding this feature to your main branch?

Best,
Charles

user: cvanwynsberghe
issue_6821: state: open
title: information ratio criterion for trees?
body: I was just playing around with rapidminer and they have an additional splitting criterion for decision trees, which is also the default. Using their default settings, I get a tree of depth 2 that is as good as our tree of depth 8 or something. I think we should check out the splitting criterion.

user: amueller
issue_6806: state: open
title: [WIP] Robust PCA algorithm
body: Addressed #5851 
This is an implementation of the Robust PCA algorithm as described in 
http://statweb.stanford.edu/~candes/papers/RobustPCA.pdf

Thanks to @amueller for his patience and helping me out with this. I would appreciate any suggestions on a possible example. Can we write a function to download the escalator dataset and run the algorithm on that as shown in the paper ?

Also, I am not sure if my implementation of `transform` and `inverse_transform` is correct. I am doing something similar to what `PCA` is doing with the exception of adjusting for mean.  @amueller Am I missing something ? Ideally Line 26 in `test_rpca.py` should not fail, isn't it ?

user: vighneshbirodkar
issue_6777: state: open
title: Perplexity not monotonically decreasing for batch Latent Dirichlet Allocation
body: When using the batch method, the perplexity in LDA should be non-increasing in every iteration, right?
I have cases where it does increase. If this is indeed a bug, I'll investigate.

user: amueller
issue_6725: state: open
title: Normalization in SVD solver for LinearDiscriminantAnalysis can produce large scaling coefficients
body: #### Description

The SVD solver contains the following code

``` python
std = Xc.std(axis=0)
# avoid division by zero in normalization
std[std == 0] = 1.
fac = 1. / (n_samples - n_classes)
#2) Within variance scaling
X = np.sqrt(fac) * (Xc / std)
```

This can result in very large scaling coefficients if std contains small values not exactly zero.
#### Steps/Code to Reproduce

Example:

``` python
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

lda = LinearDiscriminantAnalysis()
data = [[ 1, 2, 0, 4],
             [5, 6, 1e-7, 8],
             [9, 10, 0, 12]]
labels = [0,0,1]

lda.fit(data, labels)
print lda.scalings_
```

```
array([[  8.83883476e-02],
       [  8.83883476e-02],
       [  3.53553391e+06],
       [  8.83883476e-02]])
```

The scaling value resulting from the normalization could ( and does ) result in poor classification performance. I haven't as yet checked this with other implementations to compare the results.
#### Versions

Windows-7-6.1.7601-SP1
('Python', '2.7.11 |Anaconda 2.5.0 (32-bit)| (default, Mar  4 2016, 15:18:41) [MSC v.1500 32 bit (Intel)]')
('NumPy', '1.10.4')
('SciPy', '0.17.0')
('Scikit-Learn', '0.17')

user: planetmarshall
labels: Needs Decision: Requires decision
issue_6723: state: open
title: LabelBinarizer single label case
body: 1. It should be documented that in case of a single label, doing a `fit_transform` returns an array of negative labels, i.e the majority class is set to `neg_label` by default.
2. Also
   
   ```
   lb = LabelBinarizer()
   lb.fit([0])
   lb.transform(["p"])
   array([[0]])
   ```

this seems to be a bug to me.

user: MechCoder
issue_6656: state: open
title: var.monotone option in GradientBoosting
body: Hi, is it possible to add the equivalent of the var.monotone option in R GBM package to the GradientBoostingClassifier/Regressor? Sometimes it is really useful when we know/want some factors to have monotonic effect to avoid overfitting and non-intuitive results.

Thanks!

user: jingsliu07
issue_6645: state: open
title: Why is KNearestNeighbors slower with n_jobs > 1?
body: Hi all! I've asked this on StackOverflow but I didn't get any answer :(. I can't understand why KNearestNeighbors is slower with n_jobs > 1. I've set the start method for multiprocessing to _forkserver_ but it's near 100 times slower...

```
>>> import platform; print(platform.platform())
Linux-4.4.5-1-ARCH-x86_64-with-arch-Arch-Linux
>>> import sys; print("Python", sys.version)
Python 3.5.1 (default, Mar  3 2016, 09:29:07) 
[GCC 5.3.0]
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.11.0
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 0.17.0
>>> import sklearn; print("Scikit-Learn", sklearn.__version__)
Scikit-Learn 0.17.1
>>> 
```

I run this sample code and I post results.

``` python
#!/usr/bin/env python

import numpy as np
from sklearn import neighbors
from time import time
from sklearn import datasets
import multiprocessing as mp

if __name__ == "__main__":
    mp.set_start_method('forkserver', force = True)
    np.random.seed(123456)

    iris = datasets.load_iris()

    knn = neighbors.KNeighborsClassifier(n_neighbors = 3, n_jobs = 2)

    perm = np.random.permutation(iris.target.size)
    iris.data = iris.data[perm]
    iris.target = iris.target[perm]
    knn.fit(iris.data[:100], iris.target[:100])

    start = time()
    score = knn.score(iris.data[100:], iris.target[100:])
    end = time()

    print(score)
    print(end-start)
```

Output:

```
[antonio@Antonio-Arch Práctica 1: Búsquedas por Trayectorias]$ python cv.py 
0.94
0.001409292221069336 # Time with n_jobs = 1
[antonio@Antonio-Arch Práctica 1: Búsquedas por Trayectorias]$ python cv.py 
0.94
0.10267972946166992 # Time with n_jobs = 2
```

Could anyone explain me this behaviour? I've tried with so much complex datasets and the behaviour is the same :(. 

Thank you!

user: 29antonioac
labels: Needs Decision: Requires decision
issue_6630: state: open
title: Different alpha selection strategies in LinearModelCV
body: LinearModelCV selects the alpha with the minimum MSE across folds:

```
for l1_ratio, l1_alphas, mse_alphas in zip(l1_ratios, alphas,
                                                   mean_mse):
    i_best_alpha = np.argmin(mse_alphas)
```

This is not always advisable, ie we might instead want to trade a slightly "worse" alpha for a less complex model (less variables), or even a more complex one (in my case for example I'd be happy to trade a higher MSE for a few more variables selected).

[Glmnet has the option](https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html) to use either the min-MSE alpha, or a less complex one within 1 standard deviation of the minimum (they call it lambda.1se).

We could add an argument taking three values: "alpha.min", "alpha.1se+", "alpha.1se-" or something similar, to indicate three possible alpha selections:
- minimum, like now
- least complex within 1 se of min
- most complex within 1 se of min

Or even use an argument whose default value is the function `np.argmin`, so that the user can specify his own alpha-selection function (still limited to only mse_alphas, though).

What do you think?

user: fabio-t
issue_6604: state: open
title: [MRG] LabelPropagation and LabelSpreading enhancements
body: The following enhancements have been made:
- `X` accepts sparse matrixes
- `X` can now directly represent an adjacency matrix. This is useful when X represents a graph and therefore no kernel is required (`kernel=None`)
- Currently if a sample has been predicted with 0 or NaN probability the argmax will select the first column and therefore classify the sample with the class with an index of 0. e.g. `argmax([0,0,0]) = 0` This is obviously undesirable. Therefore now such samples will remain unclassified with a class of -1
- A `transduction_prob_` attribute has been added after fitting which contains the probability of label assigned to each sample via the transduction

Note:
- I don't know how to implement `predict_proba` and `predict` when `kernel=None`. Currently a `NotImplementedError` is raised. Any suggestions on how to do it is much welcomed

user: joshlk
labels: Needs Decision: Requires decision
issue_6598: state: open
title: [MRG] Stratifiedkfold continuous (fixed)
body: this PR addresses the issue #4757

tests attached (fixed) 

user: DSLituiev
issue_6568: state: open
title: Balanced/Weighted Sampling 
body: Many classification applications need to deal with skewed input data - recently for several projects I've had to implement techniques to re-weight samples during training to get the best results - this can ideally be supported generically by scikit-learn in https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/cross_validation.py

In my use case I was able to get significantly better results by assuming a uniform prior during training with the skewed labels- but it makes sense to have a generic way to add weights to the sampled training distribution for cases where researchers have good reason to incorporate a certain prior. 

user: anjishnu
issue_6558: state: open
title: [MRG]Added possibility of adding both values and proportions in export_graphviz
body: As described in #6553 it would be great to have such ability. This pull request adds such ability + some tests for export graphviz.

user: zuku1985
labels: Needs Decision: Requires decision
issue_6553: state: open
title: export_graphviz function for tree should give ability to export both values and proportions in each node
body: For now there are two choices - proportions or values, controlled by boolean proportion parameter. It would be helpful in some cases to see both of them on visualized tree.

user: zuku1985
labels: Needs Decision: Requires decision
issue_6540: state: open
title: [WIP] Add ensemble selection algorithm
body: This PR is an implementation of ensemble selection which is discussed in #6329 , 
however, the API of this module is not yet determined.

**Check List:**
- [x] Implement main algorithm
- [x] Add example code
- [ ] Determine API
- [ ] Test
- [ ] Detail documentation

**Reference:**
[Ensemble Selection from Libraries of Models](http://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml04.icdm06long.pdf)  R. Caruana et.al. ICML 2004

user: yenchenlin
issue_6534: state: open
title: [MRG] Add Information Gain and Information Gain Ratio feature selection functions
body: The commit implements Information Gain [1] and Information Gain Ratio functions used for feature selection. The functions are commonly used in the filtering approach to feature selection in tasks such as text classification ([2] and [3]). IG is implemented in WEKA package.

The input parameters and output values as well as tests of the functions follow the example for the chi-square function.

The coverage of sklearn.feature_selection.univariate_selection is 98%.

PEP8 and PyFlakes pass.

```
References:
-----------
.. [1] J.R. Quinlan. 1993. C4.5: Programs for Machine Learning. San Mateo,
CA: Morgan Kaufmann.

.. [2] Y. Yang and J.O. Pedersen. 1997. A comparative study on feature
selection in text categorization. Proceedings of ICML'97, pp. 412-420.
http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.32.9956

.. [3] F. Sebastiani. 2002. Machine Learning in Automatic Text
Categorization. ACM Computing Surveys (CSUR).
http://nmis.isti.cnr.it/sebastiani/Publications/ACMCS02.pdf
```

user: vpekar
issue_6524: state: open
title: Add warm_start to liblinear solver in LogisticRegression
body: I think this has been discussed before and the new `LogisticRegressionCV` model limits the benefits of this but I believe it would speed up the path algorithm for the l1 penalty. For example in (https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/linear_model/randomized_l1.py#L369)

Either that or we could make `_fit_liblinear` accept a `w0` parameter.

user: MechCoder
issue_6496: state: open
title: Logistic regression with a probability response or with number of successes/failures) rather than binary outputs
body: Hi, 

I have aggregated data impressions and clicks by set of features. And would like to use scikit-learn Logistic regression to estimate probability to click (CTR). I.e. the response variable is the number of clicks divided by the number of impressions (#clicks/#impressions).

More details in the following question in Stack overflow ;
http://stackoverflow.com/questions/33871615/train-a-model-with-probability-response-or-number-of-successes-failures-rather

Regards,
Boris

user: borisRa
issue_6451: state: open
title: Cloning CalibratedClassifierCV with prefit base_estimator
body: I recently stumbled upon this issue, I needed to clone a `CalibratedClassifierCV` object when the latter is defined on top on a pre-fit (`cv="prefit"`) base estimator. Unfortunately, because of the semantic of the clone interface, this cannot be achieved without "unfitting" the pre-fit base estimator. 

I think this is somehow related to @jnothman long standing frozen estimator discussion. 

user: glouppe
issue_6448: state: open
title: [MRG] Support multi-threading of LibLinear L1 one-vs-rest LogisticRegression for # classes > 2
body: This branch subsumes the changes made to support larger feature sparse arrays. I don't know of a clean, simple way of separating the two without creating a new fork.
I suspect that if this change is of interest, the maintainers may want to change how the functionality is interfaced in the API, which may require a new branch anyway.

user: mannby
labels: Needs Decision: Requires decision
issue_6409: state: open
title: Split binary_tree.pxi into binary_tree.pyx + binary_tree.pxd and import rather than include
body: There is a pxi file at `neighbors/binary_tree.pxi`

I may be wrong here, but isn't importing a `pyx` rather than including the `pxi` a better/cleaner way? Or is it just the same? (If so is there a specific reason why that won't work here?)

Also the way the [current `cythonize.py`](https://github.com/scikit-learn/scikit-learn/blob/4b7a0e01cecfbc8be4b06c3798010a31bd2945e4/build_tools/cythonize.py) is setup, it will track for changes in pyx/pxd files alone and recythonize them if changed.

If the `binary_tree.pxi` is to be untouched, we need a minor hack to make it compile the `kd_tree.pyx` and `ball_tree.pyx` whenever the `binary_tree.pxi` is modified. (Also see #6254)

If the `binary_tree.pxi` can be split into `binary_tree.pyx` + `binary_tree.pxd`, I'd like to take it up as I am trying to learn cython. (For fiddling with the `sklearn/tree` module)

**cc:** @jakevdp @amueller @ogrisel @MechCoder 

user: raghavrv
labels: Needs Decision: Requires decision
issue_6353: state: open
title: LatentDirichletAllocation's components aren't normalized
body: An LDA model's `components_` isn't easily interpretable:

``` py
>>> X = np.abs(np.random.RandomState(42).randn(5, 4))
>>> from sklearn.decomposition import LatentDirichletAllocation
>>> lda = LatentDirichletAllocation(n_topics=3).fit(X)
>>> lda.components_
array([[ 0.91847978,  2.43240154,  2.14285975,  1.10793777],
       [ 2.01874248,  0.92082054,  3.05363433,  3.58789672],
       [ 0.49890124,  0.50614032,  0.47761483,  0.51191327]])
>>> lda.components_.sum(axis=1)
array([ 6.60167884,  9.58109407,  1.99456967])
```

Looks like it's just not normalized, since normalizing in-place doesn't change `transform` outputs:

``` py
>>> d_before = lda.transform(X)
>>> lda.components_ /= lda.components_.sum(axis=1)[:, np.newaxis]
>>> d_after = lda.transform(X)
>>> norm(d_before - d_after)
```

But I don't know if a consequent `partial_fit` will still work.

Follow-up to #6320.

user: larsmans
labels: Needs Decision: Requires decision
issue_6346: state: open
title: GenericUnivariateSelect has silly defaults
body: it uses `percentile` but gives the param as `1e-5` Not sure it is worth the deprecation cycle, though.

user: amueller
issue_6338: state: open
title: Adding a utility function for plotting decision regions of classifiers
body: It's been a while when @amueller asked me about moving this utility function "upstream," but I finally got around it now (https://github.com/rasbt/mlxtend/issues/8) ...

Although such a function probably less useful in real-world applications (since we typically have more than 1 or 2 features in our dataset), but I think it would be a nice utility for replacing the many repeated lines of codes in the tutorials and examples on the scikit-learn website.

So, this is a simple matplotlib-wrapping convenience function to plot a decision surface of a classifier; it looks like this:

``` python
import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.svm import SVC

# Loading some example data
iris = datasets.load_iris()
X = iris.data[:, [0,2]]
y = iris.target

# Training a classifier
svm = SVC(C=0.5, kernel='linear')
svm.fit(X,y)

# Plotting decision regions
plot_decision_regions(X, y, clf=svm, res=0.02, legend=2)

# Adding axes annotations
plt.xlabel('sepal length [cm]')
plt.ylabel('petal length [cm]')
plt.title('SVM on Iris')
plt.show()
```

![plot_decision_regions_9_0](https://cloud.githubusercontent.com/assets/5618407/12995332/9d62876c-d0f4-11e5-9a7e-73f27f275a79.png)

It also supports 1D decision regions (if the input array has only one feature column) and you can highlight the test datapoint, which can be quite useful for teaching (tutorial) purposes.
![plot_decision_regions_11_0](https://cloud.githubusercontent.com/assets/5618407/12995381/e6c34f0e-d0f4-11e5-9403-db80f6176482.png)
![plot_decision_regions_14_0](https://cloud.githubusercontent.com/assets/5618407/12995384/ec6d6e4e-d0f4-11e5-99f4-d212228bede7.png)
![plot_decision_regions_16_0](https://cloud.githubusercontent.com/assets/5618407/12995390/f5260c76-d0f4-11e5-92d2-4f50a6d7d9a4.png)

<br><br>

<hr>

I am looking forward to feedback! Also, I am wondering how (or if) we implement tests for such a function?

user: rasbt
labels: Needs Decision: Requires decision
issue_6329: state: open
title: Stacking: Add Ensemble Selection from Libraries of Models
body: The following paper by Rich Caruana
http://www.niculescu-mizil.org/papers/shotgun.icml04.revised.rev2.pdf
presents an simple greedy algorithm to stack models.

It is reported by many people to be an excellent strategy in data competitions. We should implement it. The paper has only 333 citations, which is on the low side for our criteria, but I hear a lot of good on it.

user: GaelVaroquaux
issue_6322: state: open
title: Add TimeSeriesCV and HomogeneousTimeSeriesCV
body: I get this asked about once a day, so I think we should just add it.
Many people work with time series, and adding cross-validation for them would be really easy.
The standard strategy is described for example [here](http://stats.stackexchange.com/questions/14099/using-k-fold-cross-validation-for-time-series-model-selection)

There are basically two cases: homogeneous time series (one sample every X seconds / days), or heterogeneous time series, where each sample has a time stamp.

For the homogeneous case, we can just put the first `n_samples // n_folds` in the first fold etc, so it's a very simple variation of KFold. Fixed in #6586.

For heterogeneous case, we need to get a `labels` array and split accordingly. If we cast that to integers, people could actually provide pandas time series, and they would be handled correctly (they will be converted to nanoseconds).

I remember arguing against this addition, but I changed my mind ;)

user: amueller
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_6303: state: open
title: WIP: Nearest Neighbor chaining for ward
body: See #4752 .

There is a problem right now with the reordering of the children (the large commented out block). This is necessary because without a priority queue, we don't remove clusters that are the absolute lowest distance from each other. Reordering the children is necessary if the unstructured call must match the structured one. It takes a very large amount of time (even longer than the clustering loop), so I am working on making it faster.

@GaelVaroquaux Here are some results on the nilearn dataset:

master (cache cleared)

Ward agglomeration 1000 clusters: 14.12s
Ward agglomeration 2000 clusters: 7.65s

ward-nn-chaining (cache not cleared from master run):

Ward agglomeration 1000 clusters: 36.22s
Ward agglomeration 2000 clusters: 32.11s

ward-nn-chaining (cache now cleared):

Ward agglomeration 1000 clusters: 23.23s
Ward agglomeration 2000 clusters: 24.21s

I'm hoping the low speedup from caching is due to the fact that the children are not ordered, but I don't know enough about Memory.cache() right now to investigate this further.

user: dwarrier
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_6284: state: open
title: [ENH] Provision generating missing values and add parameters to control the same
body: 1. Would it be worthwhile to add parameters to control missingness in dataset generators?
   
   I need this for benchmarking #5974. Thought this might come in handy for teaching too.
   
   Typically I would like to add `missing_rate`, `target_correlation_rate`, `feature_correlation_rate` and `missing_values`.
   
   The `target_correlation_rate` would control the extent to which the dataset is MNAR<sup>*</sup> and `feature_correlation_rate` would control the extent to which the dataset is MAR<sup>†</sup>.
   
   `target_correlation_rate` + `feature_correlation_rate` <= 1
   
   `1 - (target_correlation_rate + feature_correlation_rate)` would control the extent to which the dataset is MCAR<sup>‡</sup>.
   
   Does this sound good?
2. Either as an addition or as an alternative to **1**, could we have missing transformers with the above described params?

<hr/>

\* - Missing Not At Random (Missingness is correlated with the target)

† - Missing At Random but correlated with the other feature values.

‡ - Missing Completely At Random (No correlation with either the target or parameters)

Ping @agramfort @glouppe @GaelVaroquaux

user: raghavrv
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_6259: state: open
title: Bayesian Logistic Regression / Logistic Regression with ARD 
body: There are Bayesian Linear Regression and ARD regression in scikit, are there any plans to include Bayesian / ARD Logistic Regression?  ARD version will be really helpful for identifying relevant features.

user: AlexYurasov
labels: Move to scikit-learn-extra: This PR should be moved to the scikit-learn-extras repository
issue_6245: state: open
title: Multi-core training using liblinear and libsvm
body: Has anyone looked into supporting a multi-core training extension to liblinear and lbsvm (https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/multicore-liblinear/)?

There seems to be a patch available to do this:

https://github.com/fidlr/sklearn-openmp
http://fidlr.org/post/137303264732/scikit-learn-017-with-libsvm-openmp-support

user: mannby
issue_6240: state: open
title: Unintuitive `predict_proba` results with smoothed hinge loss
body: The [SGD docs](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn.linear_model.SGDClassifier.predict_proba) mention that _"Binary probability estimates for loss=”modified_huber” are given by (clip(decision_function(X), -1, 1) + 1) / 2."_.

I'm not sure what the contract is between `predict()` vs `predict_proba()`, but it's a little unintuitive that on the same input, `predict()` can work (return the correct classes) but `predict_proba()` return a uniform distribution over all classes (no differentiation between classes at all). This happens when _all_ scores are clipped to the same value in `predict_proba`, including the best score.

Is it worth making the documentation more explicit about this clipping and the predict/predict_proba divergence? Or is it understood from sklearn's API contracts?

user: piskvorky
labels: Needs Decision: Requires decision
issue_6236: state: open
title: Better example for agglomerative clustering
body: this example shows us is the behavior “rich getting richer” of agglomerative clustering that tends to create uneven cluster sizes. Graphs help to compare us where the behavior is more pronounced between average, ward and complete linkage strategy. Using random data helps to strengthen this notion.

user: cheral
labels: Needs Decision: Requires decision
issue_6231: state: open
title: Scoring functions don't know classes_
body: Moving the discussion with @amueller from https://github.com/pydata/patsy/issues/77#issuecomment-174335341.

Proposing to:
- add an optional 'labels' argument to `log_loss`
- add an argument to `make_scorer` to enable passing labels argument to the custom loss function

Related to this - perhaps also allow `make_scorer`'s `need_threshold = True` to be used for non-binary classification problems. Not sure why it's limited to binary. Totally meaningful for multi-class situations, where the metric might use the decision function to evaluate based on the rank order or something (for example, `SVC` doesn't have a cheap `predict_proba`, but has a cheap `decision_function`, so it would allow it to be used with rank-based metris). From `make_scorer` perspective, it just seems like a completely arbitrary restriction because it shouldn't care about what the custom loss function does with the `decision_function` output.

user: pkch
issue_6215: state: open
title: [WIP] Storing the best attributes of (non-GridSearch) CV models
body: Extension of work on #5498 

user: Sentient07
issue_6186: state: open
title: SGD classification unnecessarily slow
body: Multiclass prediction using SGD on sparse data is unnecessarily slow. It seems to be copying large arrays _on every call to `predict`, `predict_proba` etc_, which kills its performance.

I think the main culprit is `safe_sparse_dot`, which uses scipy's "CSR \* dense" routine:

``` python
def safe_sparse_dot(a, b, dense_output=False):
    if issparse(a) or issparse(b):
        ret = a * b  # <== this line here: `a` is CSR, `b` is dense clf.coef_.T
        if dense_output and hasattr(ret, "toarray"):
            ...
```

Because `b` is transposed and scipy's multiplication invokes `b.ravel()`, this is very slow (copies `clf.coef_` internally).

Keeping `clf.coef_.T` as a C-contiguous array [here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/linear_model/base.py#L252) improved the prediction performance of SGD classifier **7300x** for us (1s vs 137µs per call):

``` python
def decision_function(self, X):
    ...
    # before:
    # scores = safe_sparse_dot(X, self.coef_.T, dense_output=True) + self.intercept_

    # after quick fix: self.coef_T = np.ascontiguousarray(self.coef_.T)
    scores = safe_sparse_dot(X, self.coef_T, dense_output=True) + self.intercept_
    ...
```

The exact speedup numbers will vary, depending on `coef_` size (~the number of SGD classes and features).

This could be raised as an issue in scipy as well (I see no good reason for such inefficiency -- that `ravel()` is just too generous), but since the fix seems trivial, maybe it's worth addressing on sklearn side as well?

This is using scipy 0.16.1 and sklearn 0.17.

user: piskvorky
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_6162: state: open
title: sklearn.datasets.make_classification should have a n_targets parameter
body: `make_regression` has a `n_targets` parameter. It might be useful to have that in `make_classification` as well

user: MechCoder
issue_6160: state: open
title: [MRG] Refactor model_selection._search to include transductive estimators
body: This generalises `BaseSearchCV` as `BaseSearch` so that it can be used on unsupervised learning where cross-validation does not make sense. The approach is to use a pair of mixins, `SearchClusterMixin` and `SearchCVMixin` to add `_fit` methods appropriate to each type of search.

The existing `GridSearchCV` and `RandomizedSearchCV` now inherit `BaseSearch` and `SearchCVMixin` while the new `GridSearchCluster` and `RandomizedSearchCluster` inherit `BaseSearch` and `SearchClusterMixin`.

See #6154 for a little more background.

user: jamiebull1
issue_6154: state: open
title: Grid search without cross-validation for unsupervised clustering
body: It would be nice to have a framework similar to `GridSearchCV` and `RandomSearchCV` for assessing unsupervised clusterers. Given a scoring function (silhouette score, or any other domain-appropriate way of assessing how well a given clustering meets user requirements) this would be a useful way of quickly selecting an appropriate model.

The problem is that cross-validation is not necessarily appropriate for unsupervised clustering, and in my testing it doesn't seem to work very well. As a solution it would be good to be able to do one of:
1. Use `GridSearchCV` and `RandomSearchCV` (any subclasses of `BaseSearchCV` in fact) with a `cv` parameter of 1 indicating a single fold
2. Have a `BaseSearch` class that can be implemented as `GridSearch` and `RandomSearch` without cross-validation, and also replace `BaseSearchCV` as a superclass for `GridSearchCV` and `RandomSearchCV`, given something like a `CVMixin` to add cross-validation support.

user: jamiebull1
issue_6143: state: open
title: Better example for different agglomerative clustering methods?
body: Maybe it's me, but I feel the current example we have comparing agglomerative clustering algorithms doesn't provide a great intuition. I'm not super familiar with them, so maybe it's just my lack of understanding.
We currently have this:
http://scikit-learn.org/dev/auto_examples/cluster/plot_digits_linkage.html#example-cluster-plot-digits-linkage-py
That currently outputs pictures and timings. The timings are the same for the three methods ("average", "complete", "ward").

I guess you see that the average linkage has these two singletons, and complete has one relatively small cluster?
Maybe it would be nice to visualize the distribution of cluster sizes for each, and show that Ward creates more equally sized clusters? Maybe it would also be nice to compute ARI or a similar supervised clustering measure?
If there is an obvious clustering solution where the clusters have very different sizes, how do the results of Ward and the others differ?
What is happening on the "cluster comparison" plot in the top row? Why doesn't Ward get the circles right?

[ping @GaelVaroquaux @agramfort ]

user: amueller
issue_6120: state: open
title: Eigenface example is confusing
body: There is an example comparing decomposition methods on the olivetti faces:
http://scikit-learn.org/dev/auto_examples/decomposition/plot_faces_decomposition.html

It prints something like

```
Dataset consists of 400 faces
Extracting the top 6 Eigenfaces - RandomizedPCA...
done in 0.275s
Extracting the top 6 Non-negative components - NMF...
done in 2.067s
Extracting the top 6 Independent components - FastICA...
done in 2.375s
Extracting the top 6 Sparse comp. - MiniBatchSparsePCA...
done in 1.454s
Extracting the top 6 MiniBatchDictionaryLearning...
done in 1.190s
Extracting the top 6 Cluster centers - MiniBatchKMeans...
done in 0.291s
Extracting the top 6 Factor Analysis components - FA...
done in 0.693s
```

The "top 6" is wrong for everything but PCA, sparse PCA and FA, I think.
It's not the "top 6" nmf components. They are not ordered. And if you gave it more components, they would look very differently. Same for dictionary learning or K-Means.

We could fix the print message, but maybe it would be better to use more components for the different methods, even if we are not showing them. Not sure if that would blow up the runtime, though.

user: amueller
issue_6110: state: open
title: Fix docstring signature mismatch in cython code
body: Similar to #2062, but for cython code. I am not sure how easy it is to find those.

user: raghavrv
labels: Easy: Well-defined and straightforward way to resolve
issue_6091: state: open
title: MLPClassifier: using warm_start breaks adam algorithm?
body: When using `adam` algorithm (or `sgd` with non-constant rate schedule), the choice of `warm_start = True` and `max_iter = 1`, repeated `n` times, isn't equivalent to simply setting `max_iter = n`. This is because every time `AdamOptimizer()` is called again, so all its internal state is reset. Is this intentional or a bug? 

The reason I'm trying to do this in the first place is to stop the iteration after a certain amount of time expires - but I don't want to otherwise modify the behavior of the algorithm.

Given that my approach didn't work, will using `partial_fit()` with the same `X`, `y` instead of `warm_start = True, max_iter = 1` work the way I expected, i.e., will `n` repetitions of it be equivalent to `fit(max_iter = n)`? 

At the very least, it's worth documenting the behavior of `warm_start` and `partial_fit` more clearly.

user: pkch
issue_6085: state: open
title: [WIP] randomised block krylov svd
body: Initial implementation of randomised block krylov svd from the paper:

http://www.cameronmusco.com/personal_site/pdfs/blockKrylov.pdf

Also see this talk for more information:

http://research.microsoft.com/apps/video/default.aspx?id=259582&l=i

user: kashif
issue_6077: state: open
title: changed pls_.py to properly scale data, modified documents
body: 1.  In many instances of the variable `scale` setting was ignored.  Data was always centered, and always scaled.  The data is now scaled appropriately when `scale=True` is set, and not scaled when it's `False`
2. The return from the `predict()` function was always de-centered, but never de-scaled, so that we got:
   `Ypred = Ypred + self.y_mean_`  for all cases
   instead of:
   `Ypred = (Ypred * self.y_std_) + self.y_mean_` when `scale=True`
3. In the nipals algorithm, either C and Q matrices are used for deflation modes (regression or canonical).  This is now implemented.  The #FIXME note is now not necessary, as `coef_` is calculated differently depending on those two cases.  The calculated `coef_` was also scaled unnecesarily by standard deviations of X and Y: this is now commented out.  Only matrices W and C are scaled by their norms in the inner loop.  The outer loop scales P, Q, and C (canonical and regression) matrices by `T.T*T` or `U.T*U`
4. Some documentation was changed to reflect code modifications.  Also, instructions to use `sklearn.preprocessing.scale` are included, as that could be a useful step for most users.
5. It should be noted that the example included in the comment sections now gives better predictions of Y.
6. if X and Y are scaled using `sklearn.preprocessing.scale` (or column mean centered and column variance scaled), the pls_ functions all return the SAME results when `scale=True` and  `scale=False` this was not the case in the previous version of the code

user: OlaPaw
issue_6010: state: open
title: LatentDirichletAllocation Default Parameters
body: The LDA model doc says it defaults to alpha of 1/ n_topics

Incase of JGibbLDA, they use 50/n_topics and refer to the paper http://www.pnas.org/cgi/reprint/0307752101v1.pdf

Personally also, most implementations of LDA I have come across in literature default to a value, an order greater than 1/n_topics.

I think this needs to be looked at.

user: chiragnagpal
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_6002: state: open
title: Problems with data scaling in sklearn.cross_decomposition.PLSRegression
body: I am trying to fit some spectral data using PLS, and am having difficulties with the module.

Essentially, when I use the default value of `scale=False`, I get a prediction, BUT all my predictions are scaled, and I just cannot figure out how to convert back to my original data space.  The same is true for the example code.  As you see, the scaled prediction is just off.  I assumed that I should be able to revert back to the "unscaled" data using `pls_scaled.y_mean_` and `pls_scaled.y_std_`, but that doesn't seem to work for me.  Any suggestions would be highly appreciated.

```
Y = np.array([[0.1, -0.2], [0.9, 1.1], [6.2, 5.9], [11.9, 12.3]])
X = np.array([[0., 0., 1.], [1.,0.,0.], [2.,2.,2.], [2.,5.,4.]])
pls_not_scaled=PLSRegression(n_components=2, scale=False)
pls_scaled=PLSRegression(n_components=2, scale=True)
pls_not_scaled.fit(X,Y)
Out[]: PLSRegression(copy=True, max_iter=500, n_components=2, scale=False, tol=1e-06)
pls_scaled.fit(X,Y)
Out[]: PLSRegression(copy=True, max_iter=500, n_components=2, scale=True, tol=1e-06)
Out[]: PLSRegression(copy=True, max_iter=500, n_components=2, scale=False, tol=1e-06)
Y_not_scaled_pred = pls_not_scaled.predict(X)
Y_scaled_pred = pls_scaled.predict(X)

print(Y)
[[  0.1  -0.2]
 [  0.9   1.1]
 [  6.2   5.9]
 [ 11.9  12.3]]

print(Y_not_scaled_pred)
[[  0.11029323  -0.09323388]
 [  0.86825958   0.77077371]
 [  6.24049125   6.31999396]
 [ 11.88095594  12.1024662 ]]

print(Y_scaled_pred)
[[ 1.52568016  1.47577156]
 [ 2.43367318  2.3584298 ]
 [ 6.25638942  6.26638408]
 [ 8.88425724  8.99941456]]
```

For those with a more visual bend:

```
plt.grid('on')
plt.scatter(Y[:, 0], Y_not_scaled_pred[:, 0], color='blue')
plt.scatter(Y[:, 0], Y_scaled_pred[:, 0], color='red')
plt.xlabel('Predicted')
plt.ylabel('Measured')
plt.legend(['not_scaled','scaled'],'lower right')
```

![figure_1](https://cloud.githubusercontent.com/assets/13950307/11730049/a1fcfa66-9f60-11e5-960f-90fa260eef6a.png)

user: ghost
issue_5972: state: open
title: Stratifying Across Classes During Training in ShuffleSplit #5965
body: This is the PR reg: Stratifying Across Classes in ShuffleSplit #5965 , in which I have done the following:
Introduced a new optional flag stratify_across_classes to StratifiedShuffleSplit.
Programmed the inner-working to achieve the expected CV splits.
Enhanced the validation of the specification for train_size and test_size (sum=1).

**Also**, I feel there might be a potential bug in the ShuffleSplit class, which primarily anchors on test_size, and doesn't handle the specification of train_size (and infer the test_size appropriately). It seems like not much testing has been performed in cases when the user only specifies train_size.
### Example outputs from running the committed code:

Class sizes:  [30, 80]
test_size:  0.3
------- specifying test_size --------
With the proposed stratification:
  Training:  21 21   Testing:  9 59
Without 
  Training:  21 56   Testing:  9 24

Class sizes:  [70, 50]
test_size:  0.3
------- specifying test_size --------
With the proposed stratification:
  Training:  35 35   Testing:  35 15
Without 
  Training:  49 35   Testing:  21 15

Class sizes:  [20, 90]
test_size:  0.1
------- specifying test_size --------
With the proposed stratification:
  Training:  18 18   Testing:  2 72
Without 
  Training:  18 81   Testing:  2 9

<!-- Reviewable:start -->

---

This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/scikit-learn/scikit-learn/5972)

<!-- Reviewable:end -->

user: raamana
labels: Needs Decision: Requires decision
issue_5965: state: open
title: Stratifying Across Classes in ShuffleSplit
body: Hello sk'ers,

**Great work guys** in developing scikit-learn!

An under-appreciated and yet an important issue in machine learning is the class imbalance during the training of the classifier [1,2,3]. SVM in particular is sensitive to this [1]. Class-imbalance is not uncommon, esp. in my world of NeuroImaging datasets. Where sample sizes are small, it can be difficult to let go of data to equalize the size of different classes during training. I have overcome this by implementing a variant of repeated hold-out (implemented as [ShuffleSplit](https://github.com/scikit-learn/scikit-learn/blob/f8f0d2dbc70ec9d165082add796812195fab6af3/sklearn/model_selection/_split.py#L839) here), that stratifies the class sizes within the training set. I achieve this by choosing the size (for all the  classes) based on fixed percentage (user-chosen) of the smallest class. This has helped me in [achieving balanced sensitivity and specificity](http://www.sciencedirect.com/science/article/pii/S0197458014005521) for my predictive models, and I feel this would be a worthy inclusion in scikit-learn.

This can be achieved by having an optional flag, such as 

``` python
stratify_across_classes_in_training_set=True
```

to StratifiedShuffledSplit, which would act differently in this [place](https://github.com/scikit-learn/scikit-learn/blob/f8f0d2dbc70ec9d165082add796812195fab6af3/sklearn/model_selection/_split.py#L1060)

``` python
 n_i = np.round(n_train * p_i).astype(int)
```

Let me know what you think - I would be very happy to contribute this to ShuffleSplit (and perhaps KFold) implementations of CV. 
### References:
1. Batuwita, R., & Palade, V. (2012). CLASS IMBALANCE LEARNING METHODS FOR SUPPORT VECTOR MACHINES.
2. Visa, S., & Ralescu, A. (2005). Issues in mining imbalanced data sets-a review paper. Proceedings of the Sixteen Midwest Artificial  ….
3. Wallace, B. C., Small, K., Brodley, C. E., & Wang, L. (2011). Class Imbalance, Redux. Data Mining (ICDM).
4. Raamana, P. R., Weiner, M. W., Wang, L., & Beg, M. F. (2015). Thickness network features for prognostic applications in dementia. Neurobiology of Aging, 36, S91–S102. http://doi.org/10.1016/j.neurobiolaging.2014.05.040

user: raamana
labels: Needs Decision: Requires decision
issue_5884: state: open
title: Implement Gower Similarity Coefficient
body: As suggested by @lesshaste

Paper - http://cbio.ensmp.fr/~jvert/svn/bibli/local/Gower1971general.pdf

I can implement this if there is sufficient interest?

@jnothman @amueller @agramfort 

user: raghavrv
issue_5851: state: open
title: Robust PCA
body: I think it would be great to add robust PCA.
The original algorithm is here http://statweb.stanford.edu/~candes/papers/RobustPCA.pdf
There is an implementation here: https://github.com/dfm/pcp by @dfm

This paper using Trimmed Grassmann Average promises better performance: http://files.is.tue.mpg.de/black/papers/RGA2014.pdf
@glennq implemented it here: https://github.com/glennq/tga

It is a bit unclear to me at the moment what the benefits of the two approaches are, and how they compare for example for outlier detection.
Also, I think netflix uses an improved version of the candes algorithm, for which I didn't find a reference yet (need to ping the netflix people).

user: amueller
issue_5826: state: open
title: New estimator: Rakel
body: Hello,

I am proposing a new estimator, named Rakel\* and already discussed in the mailing list. The pdf description of the estimator can be found here: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97.5044&rep=rep1&type=pdf

It implements Rakel lazily and only supports examples with 0 and 1 in the target values.
I tested it with both Python 2 and 3 and also tested the coverage (was 100% on the additions for Python 2 without compatibility, but should have been lowered for compatibility with Python 3).

Being not totally sure about where to put the various files I used for it, i put them at the same level as multiclass.py for now, but any advice on where to move them is welcome.

For the doc, I have trouble generating the html and thus cannot check that it is absolutely correct.

In general, advices are welcome on how I should modify it to be more friendly.

Best regards.

(*)

```
    Tsoumakas, G., Vlahavas, I.:

    Random k-labelsets: An ensemble method for multilabel
    classification.

    In: Proceedings of the 18th European Conference
    on Machine Learning (ECML 2007),

    Warsaw, Poland (2007) 406-417
```

user: Al-Pena
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_5818: state: open
title: Add the Diffusion Map Dimensionality Reduction Technique
body: The diffusion map is a dimensionality reduction technique that uses a transitional probability as its "distance" measure. It is noise-resistant and non-linear. Moreover, the algorithm itself is fast and scalable (I've used it with 50k+ samples). I wrote a quick implementation [here](https://gist.github.com/rohanp/2b950640c94c8e293c40). 

It has been of interest in numerous scientific fields:
[Diffusion maps](http://www.sciencedirect.com/science/article/pii/S1063520306000546) (1119 citations) 

[Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps](http://www.pnas.org/content/102/21/7426.short)
 (672 citations)

[Functional diffusion map: a noninvasive MRI biomarker for early stratification of clinical brain tumor response](http://www.pnas.org/content/102/15/5524.short) (491 citations) 

[Diffusion maps and coarse-graining: A unified framework for dimensionality reduction, graph partitioning, and data set parameterization](http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1661543) (450 citations)

Furthermore, a Google scholar search for "diffusion map dimensionality reduction" yields more results than "isomap", "locally linear embedding", or "hessian eigenmapping". 

If I fix up my implementation, would you all be interested in having this in sklearn? 

user: rohanp
issue_5789: state: open
title: Expose base_estimator interface in isolation forest
body: As a followup of #5678, it is clear that it is very important to have a fine control over the tree structure of the base estimators in the forest. For now, the only way to tune this is through `max_samples`, which both acts on the effective number of training samples and on the `max_depth` value of the trees. I believe we should decouple this and make it possible to adjust `max_samples` without necessarily changing the tree structure, and vice-versa. 

Practically, this simply means adding `max_depth` and other tree hyper-parameters to the interface of the `IsolationForest`, and plug default `"auto"` values wherever it is fit.

user: glouppe
issue_5736: state: open
title: Introduce a solver="auto" for LogisticRegression
body: Logistic regression has many different possible solvers. Choosing the best is difficult and requires expert knowledge.

I think that it would be useful to introduce a 'solver="auto"' that tries to make a good guess on the best approach. It would have to be tuned using heaving benchmarks on many different problems.

A good place to start is using the script bench_rcv1_logreg_convergence.py in benchmarks, and applying it to a variety of problems.

Would people agree with such an idea?

user: GaelVaroquaux
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_5719: state: open
title: RandomProjections need better default n_components
body: Currently `n_components` is not data dependent. Should it always warn on low-dimensional data?
Or should we try to adjust `n_components`?

Arguably using it on low-dimensional data is a user mistake, so maybe the warning is ok?
But we do use it in our examples on low-dim data....

user: amueller
issue_5685: state: open
title: Aim to raise warnings as errors in test suite
body: Going through the fun of seeing all the deprecated usages of numpy in master and 0.17, I think it would be nice to let nose break on warnings.
I don't think there should be any warnings when running the tests, and that would put the effort much more on the contributors, instead of the maintainers.

user: amueller
issue_5614: state: open
title: Ward FeatureAgglomeration:  Userwarning n_component, output is unstructured
body: When using FeatureAgglomeration with Ward linkage on a 3d dataset, I setup with adjacency matrix with grid_to_graph ( output seems fine ). However, when running the FeatureAgglomeration, this warning is issued:
"UserWarning: the number of connected components of the connectivity matrix is 158 > 1. Completing it to avoid stopping the tree early."

The 158 number is odd, but everything runs well, except that the outputted labels are unstructured (as if the connectivity matrix is not set correctly). 
## My code:

shape=mask.shape  //  ( 3d image of 101681 non-zero voxel, 96 x 114  x 96)
data = masker.fit_transform(X)    //   (list of 3d image  63 times 96 x 114  x 96)

connectivity = image.grid_to_graph(n_x=shape[0], n_y=shape[1],
                                       n_z=shape[2], mask=mask)
print connectivity.nnz     // gives 549509 non-zero connections

ward = FeatureAgglomeration(n_clusters=500, connectivity=connectivity,
                                linkage='ward', memory='nilearn_cache')
ward.fit(fmri_masked)

---

OUTPUT:
"UserWarning: the number of connected components of the connectivity matrix is 158 > 1. Completing it to avoid stopping the tree early."

user: michaelbernier
issue_5609: state: open
title: MAINT: unified treatment of MACHINE_EPSILON and np.finfo in the code base
body: `np.finfo` is used often in the code base to judge numerical imprecisions or to write sensible stopping criteria.

```
gpatrini@is149509:~/dev/scikit-learn(t-sne-zero-var)$ git grep 'np.finfo' | wc -l
61
git grep '_EPSILON' | wc -l
40
# Probably some more ...
```

It may be nice to have a uniform way to refer to those numbers, better than a case by case approach. This may help new contributors too. Perhaps to add to `utilities.rst`.

user: giorgiop
labels: Refactor: Code refactor
issue_5602: state: open
title: BUG: StandardScaler partial_fit overflows
body: The recent implementation of `partial_fit` for `StandardScaler` can overflow. A use case there is to transform indefinitely long stream of data, but that is problematic with the current implementation. The reason is that to compute the running mean, [we keep track](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/extmath.py#L788) of the sample sum.

Here the code to reproduce the behavior. To simulate long stream of data would take long time; instead, I use samples with very large norm but the effect is the same. The same batch is presented to the transformer many times. The mean should be same.

```python
from sklearn.preprocessing import StandardScaler
import numpy as np

rng = np.random.RandomState(0)

def gen_1d_uniform_batch(min_, max_, n):
    return rng.uniform(min_, max_, size=(n, 1))

max_f = np.finfo(np.float64).max / 1e5
min_f = max_f / 1e2
stream_dim = 100
batch_dim = 500000
print("mean overflow: batch vs online on %d repetitions" % stream_dim)

X = gen_1d_uniform_batch(min_=min_f, max_=min_f, n=batch_dim)

scaler = StandardScaler(with_std=False).fit(X)
print(scaler.mean_)
[  1.79769313e+301]

iscaler = StandardScaler(with_std=False)
batch = gen_1d_uniform_batch(min_=min_f, max_=min_f, n=batch_dim)
for _ in range(stream_dim):
    iscaler = iscaler.partial_fit(batch)
RuntimeWarning: overflow encountered in add
  updated_mean = (last_sum + new_sum) / updated_sample_count

print(iscaler.mean_)
[ inf]
```

user: giorgiop
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_5593: state: open
title: [MRG] ENH: Support threshold='auto' in Birch
body: Use `estimate_bandwidth` from MeanShift to provide an optional `threshold='auto'` parameter for Birch.

Meanwhile, it would be good if we are able to set a heuristic for `n_samples` so that we avoid calculating the nearest neighbors for all samples.

user: MechCoder
issue_5585: state: open
title: ENH: interface for SVD solvers
body: We could make an interface for the different SVD solvers available. See https://github.com/scikit-learn/scikit-learn/pull/5299#issuecomment-150626913. See also #5545.

user: giorgiop
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_5532: state: open
title: [WIP] Add `return_std` option to ensembles
body: This supersedes #3645.
- [x] Implement http://arxiv.org/pdf/1311.4555.pdf for confidence intervals in RFs
- [ ] Update GBRT's API for quantile regression
- [x] Update and polish example
- [ ] Add tests
- [ ] Add warnings in GPs if fixed parameters and n_restarts > 0

user: glouppe
labels: Needs Decision: Requires decision
issue_5518: state: open
title: [BUG?] CalibratedClassifierCV is not marked as a meta-estimator
body: ``` python
from sklearn.utils.testing import all_estimators
estimator_names = zip(*all_estimators(include_meta_estimators=False))[0] 
'CalibratedClassifierCV' in estimator_names
```

user: ainafp
issue_5516: state: open
title: Adding Fall-out, Miss rate, specificity as metrics
body: It would be nice if these rates would be included in the metrics module:
- False positive rate (Fall-out)
- False negative rate (Miss rate)
- True negative rate (specificity)

user: languitar
issue_5501: state: open
title: [WIP] Infomax ica
body: Fixes #3379 

Still to do:
- Examples
- Documentation

user: KamalakerDadi
issue_5373: state: open
title: Kernel K-means
body: Several people including @NelleV  seem to be interested in merging my kernel K-means code in scikit-learn:
https://gist.github.com/mblondel/6230787

I am open to it if someone wants to work on it. This will of course need docs, tests and examples.

Note that this implementations requires the kernel matrix to fit in memory but this also the case of Kernel PCA and Kernel Ridge in scikit-learn.

user: mblondel
issue_5327: state: open
title: sklearn.cluster.AgglomerativeClustering: Can we do without completing the matrix? 'UserWarning:the number of connected components of the connectivity matrix is *>1. Completing it to avoid stopping the tree early.' 
body: I have tried this both on the latest 0.16.1 version and on the latest bleeding edge version of sklearn '0.17.dev0' and this appears to be an issue in both.

I use `sklearn.cluster.AgglomerativeClustering(affinity='precomputed',connectivity=Cmat,linkage='complete')`
where Cmat is a connectivity matrix in which there are disconnected components. 
As indicated by the source code, I get the error message UserWarning: the number of connected components of the connectivity matrix is *>1. Completing it to avoid stopping the tree early. 

However, reading the source code I see that when completing the connectivity matrix the developers are wondering whether the clustering can take place without completing the matrix:
""XXX: Can we do without completing the matrix?""

I am interested exactly in this development. Do you think sklearn is planning to fix this and make it possible to do the clustering without completing the matrix? I think it should not be too hard.

Best,
Zhana

user: zkuncheva
issue_5318: state: open
title: types in dataset bunches are inconsistent
body: I just saw that iris.feature_names is a list but iris.target_names is an array. That is odd. Not sure how it is for the other datasets.
This is pretty minor but some consistency would be nice.

user: amueller
labels: Needs Decision: Requires decision
issue_5315: state: open
title: Make Lars and LassoLars parallel on the multiple targets
body: Lars and LassoLars (in least_angle.py) could have the loop on the number of targets made parallel (using joblib.Parallel, as everywhere in the codebase).

user: GaelVaroquaux
issue_5312: state: open
title: [MRG+1] check_X_y should copy y also if copy is set to be True
body: Is it acceptable to expect y also to be copied when `copy` is set to True.

This makes sure that y is not modified when y is centered inplace. I hit this bug here, https://github.com/scikit-learn/scikit-learn/pull/5291/files#diff-7416ccedd45a5840c67ff7877d24e1ceR52

I also added a test case that fails in master.

user: MechCoder
issue_5287: state: open
title: [MRG] Implement FABIA biclustering algorithm
body: This PR adds the FABIA biclustering algorithm to sklearn. Fabia is a general biclustering algorithm based on matrix factorization.

I had discussed this addition with Kemal Eren (who implemented the rest of the biclustering package during the GSoC 2013) way back in 2013. As far as I know, his mentors (IIRC @GaelVaroquaux )  welcomed the idea back then. I've had the code lying around in #2476 , but never got around to polish it until now. If there is still interest in the addition, here it is.

Notes: I also have a Cython version that is ~20-200% faster (the larger the amount of biclusters to detect, the smaller the speedup. In realistic settings it will be close to 20%). However that implementation is based around the `cython_lapack` module that comes with scipy 0.16. Without that, it would require pulling a large-ish amount of lapack into sklearn. I didn't think that was worth it.

user: untom
labels: Needs Decision: Requires decision
issue_5244: state: open
title: add randomized_svd as additional solver in IncrementalPCA
body: Following discussions in #5141, we may want to add `randomized_svd` as an alternative solver to `IncrementalPCA`. In theory, this should speed up the decomposition batch by batch.

This may be the default mode using a `svd_solver='auto'`, that the user can change to `'full'` in case higher precision in required.

user: giorgiop
issue_5212: state: open
title: [RFC] Tree module improvements
body: I am planning on submitting several PRs in an attempt to merge #5041 in slowly, with the ultimate goal being a clean implementation of multithreaded decision tree building so that Gradient Boosting can be faster. With one of the main concepts merged (#5203), here is a list of separate PRs which I'd like to merge in the near future.
- [x] Reorganize _tree.pyx into several files (see PR #5230)[merged]
- [x] Add proxy impurity improvement methods to both Gini and entropy (see PR #5233)[closed]
- [x] Reevaluate constant feature caching [closed]
- [x] Support sparse data for gradient boosting (see PR #5252)
- [ ] Add caching of computation between different split levels to avoid recomputation
- [x] Ensure feature importance converge in ensemble (see PR #5261)
- [ ] Add tests to ensure the correctness of impurity values, wrt hand-computed values on toy data.

Longer range goals which I'd like to work towards (but have no clear plan as of right now) are the following:
- [ ] Add an approximate splitter 
- [ ] Add multithreading support for single decision trees
- [ ] Add a partial fit method for tree building
- [ ] Support categorical variables
- [ ] Support missing values

At this point, it will be clearer to me what specific changes to Splitter, Criteria, and TreeBuilder need to be added to make multithreading a possibility. @glouppe @arjoly @GaelVaroquaux @pprett if you have any comments, I'd love to hear them.

user: jmschrei
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_5150: state: open
title: Dense svm and zeroed weight for samples of entire class
body: This bug appears in current master, and for any dense svm class.

```
import numpy as np
from sklearn.svm import SVC
X = np.array([[0, 0, 0],
              [0, 0, 1],
              [0, 1, 0],
              [0, 1, 1],
              [1, 0, 0],
              [1, 0, 1],
              [1, 1, 0],
              [1, 1, 1]])
y = np.array([0, 0, 0, 1, 1, 1, 2, 2])
w = np.array([1, 1, 1, 1, 1, 1, 0, 0])


f = SVC(kernel='linear', probability=True, random_state=1)
f.fit(X,y, w)
print(f.classes_)
print(f.predict_proba(X))
```

Output:

```
[0 1 2]
warning: class label 2 specified in weight is not found
[[ 0.28963492  0.71036508]
 [ 0.39180833  0.60819167]
 [ 0.28963492  0.71036508]
 [ 0.39180833  0.60819167]
 [ 0.57544014  0.42455986]
 [ 0.68293573  0.31706427]
 [ 0.57544014  0.42455986]
 [ 0.68293573  0.31706427]]
```

Here we see that svmlib internally have lost 2nd class, at the same time sklean's wrapper class keeps all class labels inside, that's why predict_proba returns matrix of shape (n_samples, 2) instead of (n_sample, 3) (what is expected by bagging classifier implementation). I understand that it's insane usage of weights by itself, but together with bagging and dataset with many labels, bagging randomly zeroes complete classes, and this bug shows itself, because bagging expects that svm's return probability of classes which they hold (e.g. all classes).

I investigated this a little bit, and can try to fix this, if someone will say that all this usage with bagging makes sense (Because i don't really sure about this).

user: olologin
issue_5123: state: open
title: [Discussion] Sparse datastructure for MultinomialNB feature_count_
body: When `n_features` gets big, `feature_count_` has a large memory footprint. When using MultinomialNB with sparse features (e.g. a HashingVectorizer over short sentences), this is appears to be an unnecessary overhead. This commit adds a optional sparse argument to the constructor of MultinomialNB and the necessary adjustments to `(partial_)fit()`. A new nosetest is added, `make` runs succesfully.

`bench_sparse_mnnb.py` contains an experiment comparing the sparse v/s non-sparse implementation. Memory_profile result indicates that 300mb less memory is used, with `n_features = 2 ** 20`. Yappi indicates a 6x speed increase. 

This pull request is a work in progress, please note the #TODO's. Feedback is welcome :).

Relevant profiling logs:

memory_profile:

```
Line #    Mem usage    Increment   Line Contents
================================================
    29  349.762 MiB    0.000 MiB   @profile
    30                             def fit_set_sp(X, clf, y): #Sparse 
    31                                 clf.fit(X, y)
    32                                 y_pred = clf.predict(X)
    33  369.879 MiB   20.117 MiB       return y_pred

Line #    Mem usage    Increment   Line Contents
================================================
    22   29.664 MiB    0.000 MiB   @profile
    23                             def fit_set(X, clf, y): #Not sparse
    24                                 clf.fit(X, y)
    25                                 y_pred = clf.predict(X)
    26  349.762 MiB  320.098 MiB       return y_pred
```

Yappi pstat log:

```
Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.022    0.022    1.362    1.362 bench_sparse_mnnb.py:1(<module>)
...
        1    0.000    0.000    0.388    0.388 bench_sparse_mnnb.py:22(fit_set)
...
        1    0.000    0.000    0.064    0.064 bench_sparse_mnnb.py:29(fit_set_sp) # Sparse
...
```

user: basveeling
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_5097: state: open
title: Our R^2 makes odd assumptions, doesn't work with LeaveOneOut
body: ``` python
from sklearn.datasets import make_regression
from sklearn.cross_validation import cross_val_score, LeaveOneOut
from sklearn.linear_model import Ridge

X, y, coef_ = make_regression(random_state=42, noise=1, n_samples=200, coef=True)
cross_val_score(Ridge(), X, y, cv=LeaveOneOut(len(X)))
```

> array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
>         0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
>         0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
>         0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
>         0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])

Maybe the single sample in the LOO is not interpreted correctly? :-/

user: amueller
issue_5088: state: open
title: Problem with fetch_20newsgroups_vectorized
body: I was using the 20 newsgroups dataset on a custom estimator and the accuracy was really crappy. It took me a while to realize that the problem was not in my estimator but in `fetch_20newsgroups_vectorized`. I already had the data serialized on disk and it used to work fine, as far as I remember. I deleted the data to force a rebuild and now it works fine.

Were there changes in the loader or in joblib recently that could explain this?

If we find the change that caused this, we could possibly force a rebuild for when the data was serialized prior to this change.

user: mblondel
issue_5082: state: open
title: Setting search parameters on estimators
body: The underscore notation for specifying grid search parameters is unwieldy, because adding a layer of indirection in the model (e.g. a `Pipeline` wrapping an estimator you want to search parameters on) means prefixing all corresponding parameters.

We should be able to specify parameter searches using the estimator instances. The interface proposed by @amueller at https://github.com/scikit-learn/scikit-learn/issues/4949#issuecomment-127289568 (and elsewhere) suggests a syntax like:

``` python
char_vec = CountVectorizer(analyzer="char").search_params(n_gram_range=[(3, 3), (3, 5), (5, 5)])
word_vec = CountVectorizer().search_params(n_gram_range=[(1, 1), (1, 2), (2, 2)])
svc = LinearSVC().search_params(C=[0.001, 0.1, 10, 100])
GridSearchCV(make_pipeline(make_feature_union(char_vec, word_vec), svc), cv=..., scoring=...).fit(X, y)
```

Calling `search_params` would presumably set an instance attribute on the estimator to record the search information.

Questions of fine semantics that need to be clarified for this approach include:
1. does a call to `search_params` overwrite all previous settings for that estimator?
2. does `clone` maintain the prior `search_params`?
3. should this affect the search space of specialised CV objects (e.g. `LassoCV`)

Questions of functionality include:

a) is `RandomizedSearchCV` supported by merely making one of the search spaces a `scipy.stats` rv, making some searches `GridSearchCV`-incompatible?
b) is there any way to support multiple grids, as is currently allowed in `GridSearchCV`?

I have proposed an alternative syntax that still avoids problems with underscore notation, and does not have the above issues, but is less user-friendly than the syntax above:

``` python
char_vec = CountVectorizer(analyzer="char")
word_vec = CountVectorizer()
svc = LinearSVC()
param_grid = {(char_vec, 'n_gram_range'): [(3, 3), (3, 5), (5, 5)],
              (word_vec, 'n_gram_range'): [(1, 1), (1, 2), (2, 2)],
              (svc, 'C'): [0.001, 0.1, 10, 100]}
GridSearchCV(make_pipeline(make_feature_union(char_vec, word_vec), svc),
             param_grid,
             cv=..., scoring=...).fit(X, y)
```

Here, parameters are specified as a pair of (estimator, parameter name), but they are constructed directly as a grid and passed to `GridSearchCV`/`RandomizedSearchCV`

user: jnothman
issue_5080: state: open
title: Polymorphic clone
body: `sklearn.base.clone` is defined to reconstruct an object of the argument's type with its constructor parameters (from `get_params(deep=False)`) recursively cloned and other attributes removed.

There are cases where I think the One Obvious Way to provide an API entails allowing polymorphic overriding of clone behaviour. In particular, my [longstanding implementation](https://github.com/jnothman/scikit-learn/tree/remember) of wrappers for memoized and frozen estimators relies on this, and I would like to have that library of utilities not depend on a change to `sklearn.base`. So we need to patch the latter.

Let me try to explain. Let's say we want a way to freeze a model. That is, cloning it should not flush its fit attributes, and calling `fit` again should not affect it. A syntax like the following seems far and away the clearest:

``` python
est = freeze_model(MyEstimator().fit(special_X, special_Y))
```

It should be obvious that the standard definition of `clone` won't make this operate very easily: we need to keep more than will be returned by `get_params`, unless `MyEstimator().__dict__` becomes a param of the `freeze_model` instance, which is pretty hacky.

Alternative syntax could be class decoration (`freeze_model(MyEstimator)()`) or mixin (`class MyFrozenEstimator(MyEstimator, FrozenModel): pass`) such that the first call to `fit` then sets a frozen model. These are not only uglier, but encounter the same problems.

Ideally this sort of estimator wrapper should pass through `{set,get}_params` of the wrapped estimator without adding underscored prefixes (not that this is so pertinent for a frozen model, but for other applications of similar wrappers). It should also delegate all attributes to the wrapped estimator. Without making a mess of `freeze_model.__init__` this is also not possible, IMO, without redefining `clone`.

So. Can we agree:
- that it would not be a Bad Thing to allow polymporphism in cloning?
- on a name for the polymorphic clone method: `clone` or `clone_params` or `sklearn_clone`
  ?

user: jnothman
issue_5058: state: open
title: Inconsistent output shapes for Regressors
body: We have inconsistent output shapes for regressors when `y.ndim == 2`.
We support but warn on `y.shape == (n_samples, 1)` (consistently after #5057).
But the output shape is sometimes `(n_samples, 1)` and sometimes `(n_samples,)`.

I'm actually not sure if we are even consistent if `y.shape == (n_samples,)` :-/

user: amueller
labels: Hard: Hard level of difficulty
Breaking Change: Issue resolution would not be easily handled by the usual deprecation cycle.
issue_5055: state: open
title: SelectKBest default and parameter name
body: The default parameter of features to select in SelectKBest is 10.
This is not a reasonable default. Maybe it should be 10%? Or 50%? But 10 leads to it crashing when there are less then 10 features. And it rarely seems reasonable.

Also, the name is `k`. Usually we don't like single letter names. Maybe it should be `n_selected`? I don't have a good idea really :-/

user: amueller
issue_5004: state: open
title: Add table / dictionary for good parameter values
body: I feel we should add good grids to search over for all estimators, both in the docs and as a dict.
I think a good source would be https://github.com/automl/auto-sklearn
and possibly carret.

user: amueller
issue_4982: state: open
title: Custom indices for Nystroem approximation and other kernel methods
body: Hi, 

It would be really helpful if one could use a priori chosen landmark points for the Nystroem approximation and other kernel methods. For example, in:

[kernel_approximation.py](https://github.com/scikit-learn/scikit-learn/blob/9d9e99a27e93ad4ebbc4de397a5b74b441cbec48/sklearn/kernel_approximation.py)

in lines 470-472, these are chosen by some random permutation technique:

inds = rnd.permutation(n_samples)
basis_inds = inds[:n_components]
basis = X[basis_inds]

It would be great if one could pass to the function fit/fit_transform a variable like "basis" that is an L x N matrix, with N the number of features and L the number of landmark points, that would serve as the "basis" that in Line 472 is currently generated randomly.. I will modify kernel_approximation.py myself to do it for now, but this might be a useful feature to other users of SciKit as well. 

Thanks in advance for any comments/suggestions you might have. 

Cheers!
Evan

user: evanfeinberg
issue_4974: state: open
title: Weighted pls -- adding support for sample_weight option.
body: Added support for sample_weight option, refactoring some code along the way. See commit message for more details.

I have tested that results do not change when no sample_weight option is specified, and when sample_weight = ones(len(X)). Also, in my weighted real-world use case, results look reasonable and are an improvement over using no weights.

Deepak

user: Fenugreek
issue_4871: state: open
title: has_fit_parameter will not work with **kwargs
body: Bagging and boosting use `sklearn.utils.validation.has_fit_parameter` to produce a clear error when `sample_weight is not None` and the base estimator does not support `sample_weight`. However, the method of checking the argspec will not work if `sample_weight` is supported as a `**kwargs` in `fit`, thus raising an exception when none applies. (A similar concern exists for the proposed alternative to `sample_weight` as keyword arg, #4696.)

user: jnothman
issue_4834: state: open
title: [RFC] API / objects for out of core fitting, partial_fit pipelines
body: @GaelVaroquaux and me recently discussed better support for out-of-core processing.
This issue is to collect comments and summarize discussions.

To me, the two core issues are representing streamed datasets and out-of-core pipelines.

Out-of-core pipelines should not be that hard if we only support stateless transformers. This was discussed previously (though I don't remember where).
It would also be possible with transformers implementing `partial_fit` by going over the dataset multiple times. That is not possible in a streaming / infinite dataset setting, but would work for reading from disk.

For representing streamed datasets, I think @GaelVaroquaux proposed a "conductor" object that handles the stream and passes it to the estimator. Another approach he mentioned is creating a transformer that reads files form disk. That would work if the data is already in batches on disk, by transforming file names to samples [if we can change the number of samples that might even work with a single file].
It would be cool to read csvs in chunks with Pandas.

I'm not sure we should prototype this in scikit-learn, but I think we should at least discuss it here.

Some libraries that work on similar concepts are rosetta: http://pythonhosted.org/rosetta/#examples, fuel: https://github.com/mila-udem/fuel and pescador: https://github.com/bmcfee/pescador

user: amueller
labels: Needs Decision - API: Requires decision regarding API
issue_4822: state: open
title: [MRG] ROCCH calibration method
body: Implementation of the ROCCH calibration method, as described in http://www.jmlr.org/papers/volume13/hernandez-orallo12a/hernandez-orallo12a.pdf and http://link.springer.com/article/10.1007/s10994-007-5011-0

It is implemented as an other method within the CalibratedClassifierCV class. Moreover, the documentation is expanded to include the results.

The method performs similarly to Isotonic calibration when measured by Brier loss, F1 Score or AUC. But  with an speedup of up to 1.3X

Here an example

```
In[2]: %paste

import pandas as pd
from sklearn import datasets
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import LinearSVC
from sklearn.calibration import CalibratedClassifierCV
from sklearn.cross_validation import train_test_split
from sklearn.metrics import brier_score_loss, f1_score, roc_auc_score

X, y = datasets.make_classification(n_samples=100000, n_features=20,
                                    n_informative=2, n_redundant=10,
                                    random_state=42)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.99,
                                                    random_state=42)

def test_times(method, est):
    clf =  CalibratedClassifierCV(est, cv=5, method=method)
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    y_prob = clf.predict_proba(X_test)[:,1]
    brier = brier_score_loss(y_test, y_prob, pos_label=y.max())
    f1 = f1_score(y_test, y_pred)
    auc = roc_auc_score(y_test, y_prob)
    return brier, f1, auc

res = pd.DataFrame(columns=['brier_loss','f1_score','auc_score'])
## -- End pasted text --
Backend TkAgg is interactive backend. Turning interactive mode on.
In[3]: %timeit res.loc['GNB+isotonic'] = test_times(method='isotonic', est=GaussianNB())
1 loops, best of 3: 823 ms per loop
In[4]: %timeit res.loc['GNB+sigmoid'] = test_times(method='sigmoid', est=GaussianNB())
1 loops, best of 3: 776 ms per loop
In[5]: %timeit res.loc['GNB+rocch'] = test_times(method='rocch', est=GaussianNB())
1 loops, best of 3: 752 ms per loop
In[6]: %timeit res.loc['LSVC+isotonic'] = test_times(method='isotonic', est=LinearSVC())
1 loops, best of 3: 416 ms per loop
In[7]: %timeit res.loc['LSVC+sigmoid'] = test_times(method='sigmoid', est=LinearSVC())
1 loops, best of 3: 367 ms per loop
In[8]: %timeit res.loc['LSVC+rocch'] = test_times(method='rocch', est=LinearSVC())
1 loops, best of 3: 324 ms per loop
In[9]: res['time'] = [823, 776, 752, 416, 367, 324]
In[10]: print res
```

| brier_loss | f1_score | auc_score | time |
| --- | --- | --- | --- |
| GNB+isotonic | 0.098487 | 0.854058 | 0.939305 |
| GNB+sigmoid | 0.108937 | 0.866259 | 0.937025 |
| GNB+rocch | 0.098752 | 0.851963 | 0.939027 |
| LSVC+isotonic | 0.099401 | 0.864936 | 0.936852 |
| LSVC+sigmoid | 0.098603 | 0.862041 | 0.937657 |
| LSVC+rocch | 0.099584 | 0.864046 | 0.936733 |

user: albahnsen
issue_4817: state: open
title: allow tied-spherical and tied-diagonal cv in gmm
body: i would to suggest implementing https://github.com/scikit-learn/scikit-learn/issues/454#issuecomment-3022158 : either make 'tied' a separate flag or add modes 'tied-diagonal' and 'tied-spherical' in the gmm module.

user: eyaler
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_4813: state: open
title: predict ought to have an optional threshold argument
body: sklearn classifiers have a predict. Most classifiers have a `predict_proba` or `decision_function`. One ought to be able to set a threshold on these to make a prediction.

This is useful for example in implementing asymmetric costs. Currently i can use a recall score to penalize FN in the cross-validator. But having trained the model, i dont want the old predictions, I want a new one which uses an appropriate new threshold on the classification to correspond to this risk asymmetry.

user: rahuldave
issue_4780: state: open
title: Coordinate descent convergence warnings
body: There are a bunch of convergence warnings in coordinate descent when running the test suite that I don't think were there before:

> .../scikit-learn/sklearn/linear_model/coordinate_descent.py:444: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations

Not sure if they are related to #4686 maybe?

user: amueller
labels: Meta-issue: General issue associated to an identified list of tasks
issue_4769: state: open
title: Dictionary learning is slower with n_jobs > 1
body: Setting n_jobs > 1 in MiniBatchDictionaryLearning (and in function dictionary_learning_online) leads to worse performance.

Multi processing is handled in sklearn.decompositions, function dict_learning, l 249 

``` python
    code_views = Parallel(n_jobs=n_jobs)(
        delayed(_sparse_encode)(
            X[this_slice], dictionary, gram, cov[:, this_slice], algorithm,
            regularization=regularization, copy_cov=copy_cov,
            init=init[this_slice] if init is not None else None,
            max_iter=max_iter)
        for this_slice in slices)
```

Minimal example :
https://gist.github.com/arthurmensch/091d16c135f4a3ba5580

Output n_jobs = 1

```
Distorting image...
Extracting reference patches...
done in 0.05s.
Learning the dictionary...
done in 5.12s.
Extracting noisy patches... 
done in 0.02s.
Lasso LARS...
done in 10.24s.
```

Output n_jobs == 2

```
Distorting image...
Extracting reference patches...
done in 0.05s.
Learning the dictionary...
done in 78.98s.
Extracting noisy patches... 
done in 0.02s.
Lasso LARS...
done in 6.15s.
```

Output n_jobs == 4

```
Distorting image...
Extracting reference patches...
done in 0.05s.
Learning the dictionary...
done in 83.24s.
Extracting noisy patches... 
done in 0.02s.
Lasso LARS...
done in 3.82s.
```

We can see that transform function of MiniBatchDictionaryLearning (relying on sparse_encode function) benefits from multi-processing as expected.

Dictionary learning relies on successive calls of sparse_encode function : slowness may come from this.

user: arthurmensch
issue_4768: state: open
title: Prediction interval for Random Forests
body: Hi there,

Is there any plan to add prediction interval (range prediction) to the RandomForecastRegressor library?  I think that will be a super useful (and necessary) addition! 

Thanks,
Chen

user: pengparic01
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_4759: state: open
title: Classifier chains & Homer algorithm for multilabel classification
body: Hi

Is there any plan to add more multi-label classifiers other than [OneVsRestClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multiclass.OneVsRestClassifier.html#sklearn.multiclass.OneVsRestClassifier)?

Here is a list of few:
1. CC&ECC: [Classifier chains for multi-label classification](http://link.springer.com/article/10.1007/s10994-011-5256-5#page-1)
2. HOMER: [Effective and Efficient Multilabel Classificationin Domains with Large Number of Labels](http://www.researchgate.net/profile/Ioannis_Katakis2/publication/228386519_Effective_and_efficient_multilabel_classification_in_domains_with_large_number_of_labels/links/0deec51c744867b25c000000.pdf)

I am experimenting with some of those recently. If there is such plan, I can try to contribute some code.

user: xiaohan2012
issue_4757: state: open
title: extend StratifiedKFold to float for regression
body: It is important to stratify the samples according to y for cross-validation in regression models; otherwise, you might possibly get totally different ranges of y in training and validation sets. However, current `StratifiedKFold` doesn't allow float:

```

$ x=sklearn.cross_validation.StratifiedKFold(np.random.random(9), 2)                                                                                                                                        
/anaconda/envs/py3/lib/python3.4/site-packages/sklearn/cross_validation.py:417: Warning: The least populated class in y has only 1 members, which is too few. The minimum number of labels for any class cannot be less than n_folds=2.
  % (min_labels, self.n_folds)), Warning)

$ list(x)
[(array([], dtype=int64), array([0, 1, 2, 3, 4, 5, 6, 7, 8])),
 (array([0, 1, 2, 3, 4, 5, 6, 7, 8]), array([], dtype=int64))]
```

In case I may miss something, is there any reason why `StratifiedKFold` does not work properly for float?

user: RNAer
issue_4752: state: open
title: Nearest neighbor chaining for ward
body: Someone just complained to me that we should do neighbor chaining in ward and we are not:

https://en.wikipedia.org/wiki/Nearest-neighbor_chain_algorithm

I have not really looked it up, and I'm not very familiar with our implementation...

user: amueller
issue_4703: state: open
title: [MRG] RandomActivation
body: Tasks
- [x] Add doc in code
- [x] Add more weight initialization methods
- [x] Add another example

This is meant to be the first stage of the pipeline for the random neural network algorithm [1]

It fits on the input data by considering the number of features and then randomly generates an `n_features x n_activated` coefficient matrix where `n_activated` is the number of the "hidden layer" features defined by the user.

The coefficient matrix can be used to transform the input data to a different space.

[1] http://homepage.tudelft.nl/a9p19/papers/icpr_92_random.pdf

user: IssamLaradji
issue_4693: state: open
title:  making dictionary learning closer to the SparseNet algorithm
body: The dictionary learning algorithm was assuming that the norm of the filters was equal to one. By using a heuristic to control for the norm of the filters, we allow for a more equilibrated learning. The implementation is a simplification of the one used in the original paper from Olshausen.

The dictionary learning is tested in:
https://laurentperrinet.github.io/sciblog/posts/2015-05-05-reproducing-olshausens-classical-sparsenet.html
and this PR is tested in:
https://laurentperrinet.github.io/sciblog/posts/2015-05-06-reproducing-olshausens-classical-sparsenet-part-2.html

user: laurentperrinet
issue_4688: state: open
title: Gradient of Mixture of Gaussians (GMM) (and possibly, list of modes?)
body: Hi,

would be nice to have not only the probability associated to a certain point in the GMM as returned by `score(X)`, but also a similar function (`gradient(X)` ?) that returns the gradient. Is that possible?
Having a list of the [modes](http://en.wikipedia.org/wiki/Mode_%28statistics%29) would also be great!
[This article: Mode-finding for mixtures of Gaussian distributions](http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&tp=&arnumber=888716) provides some information on both the gradient and the modes.

user: stablum
issue_4660: state: open
title: Establish global error state like np.seterr
body: As discussed in #4497, having a global error/ warning level in sklearn seems like a good idea.

Possible things that we want to control:
- deprecations
- numerical instability
- convergence
- shape casts
- type conversions
- unused sample_props (as in #4497).
- unscaled features in estimators that want scaled features?
- ChangedBehaviorWarning

user: amueller
labels: Needs Decision - API: Requires decision regarding API
issue_4657: state: open
title: Request: references for plot_svm_scale_c
body: The example http://scikit-learn.org/dev/auto_examples/svm/plot_svm_scale_c.html is very helpful to understand some concepts on model selection. However, I would like to have the references for some of the stated results and in general I think the example would also benefit from it. In particular, I would like to know where I can find the results on the l2 penalization paragraph.

If you point me to the references, I would be glad to submit a pull request :-)

git blames @GaelVaroquaux @amueller @jaquesgrobler

user: fabianp
issue_4580: state: open
title: add icl to mixture.GMM
body: suggest adding icl = integrated completed likelihood [1], or its approximation icl-bic [2] which is an information criteria useful for determining the number of clusters. essentially it is: icl = bic + entropy of clustering

code to demonstrate:
_, probs = gmm.score_samples(X)
entropy = -sum(sum(prob_np.log(prob) for prob in probs))
icl = gmm.bic(X) + 2_entropy

[1] C. Biernacki, G. Celeux and G. Govaert, Assessing a Mixture model for Clustering with the integrated CompletedLikelihood, IEEE Transactions on Pattern analysis and Machine Intelligence 22 (2000), 719–725.
[2] G.F. McLachlan and D. Peel, Finite Mixture Models, John Wiley & Sons, Inc., 2000

user: eyaler
issue_4577: state: open
title: Precision Recall numbers computed by Scikits are not interpolated (non-standard)
body: Hi, 

Scikit Learn seems to implement Precision Recall curves (and Average Precision values/AUC under PR curve) in a non-standard way, without documenting the discrepancy. The standard way of computing Precision Recall numbers is by interpolating the curve, as described here:
http://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html
the motivation is to 
1. Smooth out the kinks and reduce noise contribution to the score
2. In any practical application, if your PR curve ever went up, then you would strictly prefer to set your threshold there rather than at the original place (achieving both more precision and recall). Hence, people prefer to interpolate the curve, which better integrates out the threshold parameter and gives a more sensible estimate of the real performance.

This is also what standard code for Pascal VOC does and explain this in their writeup:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.157.5766&rep=rep1&type=pdf

VL_FEAT also has options for interpolation:
http://www.vlfeat.org/matlab/vl_pr.html
and as shown in their code here: https://github.com/vlfeat/vlfeat/blob/edc378a722ea0d79e29f4648a54bb62f32b22568/toolbox/plotop/vl_pr.m

The concern is that people using the scikit version will see incorrectly reported LOWER performance, than what they might see reported in other papers.

user: karpathy
issue_4551: state: open
title: preprocessing.label docs
body: It seems to me that if using any sklearn estimators, people never have to use anything in preprocessing.label, as it is done under the hood.
If this is the case, I think we should mention that in the user guide.

user: amueller
issue_4522: state: open
title: [WIP] Metrics testing
body: There have recently popped up some issues about support for different metrics. See #4520 and #4452 for example.
I am trying to add more tests, but I am not sure I am familiar enough with the metrics / neighbors modules.
@jakevdp your help would be much appreciated.
Currently travis fails because the Jaccard distances in the trees seem to be very different from the scipy ones. I think this is because our trees cast everything to bool, while scipy uses floats.
This seems to be a pretty big issues, as the algorithm that is used might change automatically depending on the dataset!!!

user: amueller
issue_4515: state: open
title: Added new kernels for Mean Shift clustering
body: There is a [PR](https://github.com/scikit-learn/scikit-learn/pull/3026) that adds new kernels for mean shift clustering. However, the owner (@PeterJacob) of the PR seems to be inactive for a while. This PR includes the changes from the previous [PR](https://github.com/scikit-learn/scikit-learn/pull/3026).

In addition,

According to [the discussion here](https://github.com/scikit-learn/scikit-learn/pull/3026#issuecomment-73734480), I updated the RBF kernel.

I also refactored the control flow of _kernel_update function. There is now one return statement and therefore it is a bit cleaner.

user: erensezener
issue_4501: state: open
title: [MRG] Add distance threshold on Hierarchical Clustering, see #3796
body: It is now possible to define a distance_threshold that will be used as an early break criterion in Hierarchical Clustering. It is used in both ward_tree() and linkage_tree().
An error is now raised if n_cluster and distance_threshold are not set OR are both set
(I don't know yet what should be the behavior if the number of clusters AND the distance threshold
are set together).

user: vmichel
issue_4450: state: open
title: label_binarize without lexicographical ordering of classes
body: It might be nice to have a keyword argument to disable the sorting of classes for `label_binarize`. Sometimes classes have a natural ordering other than lexicographical that I want to preserve in the encoding.

user: jseabold
issue_4449: state: open
title: Closed form algorithms for semi-supervised learning
body: In conference paper [1] or its journal version [2], the authors give a closed form solution to semi-supervised learning and prove its equivalence to the iterative framework.

The current implementation in `sklearn.semi_supervised` also cited [1], but it corresponds to the iterative framework.

We could implement the closed form version, which is more elegant and definitely much faster.
### References
1. Zhou, Dengyong, et al. "Learning with local and global consistency." _Advances in neural information processing systems_ 16.16 (2004): 321-328.
2. Zhou, Dengyong, et al. "Semi-supervised learning by maximizing smoothness." _J. of Mach. Learn. Research_ (2004).
3. [Yoshua Bengio's book chapter](http://www.iro.umontreal.ca/~lisa/pointeurs/bengio_ssl.pdf)

user: bryandeng
issue_4414: state: open
title: dictvectorizer.inverse_transform() not giving desired output
body: [{'a':'1', 'b':'2'},{'a':'3', 'b':0}] gives [[a=1, a=3, b=2, b=0] ,[ 1, 0, 1, 0],[ 0, 1, 0, 1]]
inverse transform gives: [{'a=1':1, 'b=2': 1}, {'a=3': 1, 'b=0': 1 }] instead of [{'a':'1', 'b':'2'},{'a':'3', 'b':0}]

user: tejasvinee
issue_4406: state: open
title: Out of bag probability estimates for Random Forests
body: This is as much a proposal as a question:
Does it make sense to use out-of-bag samples for estimating the distributions in the leaves? That should give you a better estimate of uncertainty, right?
I think there is something like that in R, and it seems to make sense to me.
Ping @arjoly @glouppe 

user: amueller
issue_4391: state: open
title: Issue w/ tf-idf computation
body: Hi, 

I have some concerns about what is currently implemented as tf-idf in https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/text.py -- i.e., tf-idf = tf \* (idf + 1).

My concern is regarding the "+ 1". The comment in the source code says, "The actual formula used for tf-idf is tf \* (idf + 1) = tf + tf \* idf, instead of tf \* idf. The effect of this is that terms with zero idf, i.e.  that occur in all documents of a training set, will not be entirely ignored." This is indeed true -- the computation is effectively linearly interpolating between tf \* idf and tf -- however, adding one to idf is not standard practice. Even after extensive search, reading many IR and NLP papers regarding tf-idf, and consulting w/ IR and NLP researchers, I have not (yet!) encountered an instance of anyone doing this. I am therefore concerned that this is the default behavior in sklearn -- all the more so because there is no mention of this in the documentation (I had to read the source code to figure it out).

Although there are several standard variants for computing tf and for computing idf (these are actually pretty well documented on the <a href="http://en.wikipedia.org/wiki/Tf%E2%80%93idf">Wikipedia page</a>), they are always combined as tf \* idf, not tf \* (idf + 1). Granted (and taking the standard definition of idf for simplicity), you can subsume the "+ 1" into the idf score itself, giving idf + 1 = log(D / D_v) + 1 = log(D / D_v) + log(e) = log(e \* D / D_v) -- in effect this means you're pretending you saw e times as many documents as you actually did, and none of them contained any terms in the vocabulary (this means that all terms -- frequent and infrequent -- will be treated as if they occurred in fewer documents than they actually did). However, this is not equivalent to any of the standard variants of idf.

Since sklearn's tf-idf is widely used, I think it should reflect standard behavior/definitions in the IR and NLP literature, and thus the "+ 1" should be removed. To be clear, I'm happy for the "+ 1" to be left as an option (or even changed to be "+ c" with c speficied by the user), thereby allowing the user to linearly interpolate between tf and "standard" tf-idf, but I think users should be aware that they are doing this, rather than doing this and thinking they're using "standard" tf-idf.

user: hannawallach
labels: Needs Decision: Requires decision
issue_4357: state: open
title: Scalable Kmeans++ also known as k-means||
body: I can see Kmeans++ implementation here but do you have scalable kmeans implementation which is supposed to be better than kmeans++


**EDIT (reference)**:

Scalable K-Means++, Bahman Bahmani et al.
http://theory.stanford.edu/~sergei/papers/vldb12-kmpar.pdf

This algorithm is also known as "k-means||"
user: jagankr
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_4344: state: open
title: Fixing incoherent style of references
body: TLDR: The references in the docstrings have weird/wrong/very different formats. "There should be one-- and preferably only one --obvious way to do it."

I was looking at the reference links of some classifiers when I realized they were not formated correctly. When I was about to submit a pull request I realized that many references have a different format and according to the Zen of Python

```
There should be one-- and preferably only one --obvious way to do it.
```

Some references
- are [referencable](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html#r127), some not.
- are [not wrapped correctly](scikit-learn.org/stable/modules/generated/sklearn.neural_network.BernoulliRBM.html#sklearn.neural_network.BernoulliRBM)
- have links to the PDFs, some don't, some links are dead.

I think references should be referencable, be as concise as possible (authors, title, conference/book, etc). I'm not sure about the link to the PDFs because the URIs change over time and lead to dead links. What do you think?

My suggestion:

```
   References
    ----------

    .. [1] Hinton, G. E., Osindero, S. and Teh, Y. A fast learning algorithm
           for deep belief nets. Neural Computation 18, pp 1527-1554.
```

I'd be willing to spend a few hours on the weekend to fix the references once we decide on how it's supposed to look like.

Also, #3912 should document the format for the references.

user: sotte
issue_4306: state: open
title: sgd with warm_start learning rate
body: I think the sgd estimators should not:

```
 # Clear iteration count for multiple call to fit.
self.t_ = None
```

when the `warm_start` parameter is `True`. This way the learning will not be reset every time `fit` is "warm called".

The same applies when `fit` is called with initial intercept or coefs.

What do you think?

user: memeplex
issue_4301: state: open
title: [MRG] Choose number of clusters
body: This module implements 7 algorithms to find "optimal" number of clusters (stabiliity, gap statistic, distortion jump, silhouette, Calinsky and Harabasz index, and 2 "elbow" methods)

If a metric is needed (for example, to compute distortion, the mean distance of a point to its cluster's center), all distance of scipy.spatial.distance can be used.

user: afouchet
issue_4288: state: open
title: [WIP] Allow nan for userdefined metric in dbscan et al
body: fixes #4283

user: raghavrv
issue_4283: state: open
title: sklearn.cluster.DBSCAN: Allow for 'nan' values for user-defined metric
body: ```
import sklearn

def user_defined_metric(u, v):
    ...

cluster_algorithm = sklearn.cluster.DBSCAN(metric=user_defined_metric)
cluster_algorithm.fit(matrix)
...
```

does not work if _matrix_ contains numpy.nan values. However, the user-defined metric might be able to handle nan values.

Would be great if you could allow nan values if a user-defined metric is defined.

user: denniskl
issue_4225: state: open
title: Ensure sparse `y` is supported in grid search etc.
body: As reported at https://github.com/scikit-learn/scikit-learn/issues/1233#issuecomment-73467547, `GridSearchCV` currently breaks if it is fit with sparse `y` (at least for some supported `scipy` versions). This is due to a `len` call, when we should probably be using `check_consistent_length`. In any case, a test is needed, since we are meant to be preferring sparse matrices for multilabel data now...

user: jnothman
issue_4223: state: open
title: precision_recall_curve - assumed limits can be misleading
body: Referencing [`"The last precision and recall values are 1. and 0. respectively and do not have a corresponding threshold. This ensures that the graph starts on the x axis."`](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/ranking.py#L346-L348) 

Say you have a tie in your highest predicted probability output from the classifier with some false positives and some true positives, for example:

```
y_true = [ 1,  0,  0,  1,  0,  1,  0,  0,  0,  1,  0]
y_pred = [.9, .9, .9, .8, .8, .7, .7, .6, .5, .4, .3]
precision, recall, thresholds = precision_recall_curve(y_true, y_pred)
```

Yields:

```
Pricision: [0.400, 0.333, 0.375, 0.428, 0.400, 0.333, 1.000]
Recall:    [1.000, 0.750, 0.750, 0.750, 0.500, 0.250, 0.000]
```

![download 1](https://cloud.githubusercontent.com/assets/5210848/6098145/7fb45fec-af8a-11e4-9e21-7549071fb2b3.png)

I understand that recall goes to zero in the limit, but precision might not be so clear-cut. A _really_ difficult problem where your top prediction is a false positive would have precision go to zero in the limit I believe. For plotting with this function, this case is probably not a big deal a vertical line from 0 to 1 would be hidden by the y-axis. 

But for the drawn top-preds, as seen above, the plot becomes misleading and makes the viewer think that at least their top prediction was true positive. This may seem like a corner case, but I ran into it on a tough classification problem I was working on and was a little baffled by the output until I checked the code out.

A quick fix to preserve the intention of a clean P-R plot might be to draw a horizontal line from the highest threshold to the y-axis, this should not alter the output of most cases like [here](http://scikit-learn.org/stable/auto_examples/plot_precision_recall.html). Trying to actually calculate where it's going in the limit might be a bit overkill :-)

I would also think that adding a `1` to the end of the `thresholds` vector would be helpful when plotting both precision and recall against the probabilities.

I'm happy to open a PR if anyone thinks this is worth addressing.

user: trevorstephens
issue_4220: state: open
title: [WIP] Add warm_start to Ridge. 
body: I started working on a warm_start option for Ridge (only used for solver="sparse_cg"). However, there is an issue in the n_samples < n_features case. We need to warm-start with the dual coefficients but they are currently not returned by ridge_regression and thus not stored in the estimator. One way would be to transform the primal coefficients to dual coefficients but I couldn't find an efficient way.

See issue #769

user: mblondel
issue_4215: state: open
title: [MRG] GBM & meta-ensembles - support for class_weight
body: Refactoring #4114 on top of #4190 and removing the GBM coding style changes from the scope of this PR for ease of review. Basically I'm adding `class_weights` to the remaining ensemble classes as was done in #3961 for forests and trees.

@amueller @pprett @glouppe you have all at least glanced at #4114 so perhaps you'll have a chance to review the new refactored version here.

user: trevorstephens
issue_4211: state: open
title: RBM.partial_fit should use batches
body: As pointed out in #4205 by @treora, rbm.partial_fit should use batches (and then fit can be implemented in terms of `partial_fit`).

user: amueller
issue_4196: state: open
title: Feature request: pass meta-data per column/sample through the Pipeline
body: Dear scikit-learn community,

DISCLAIMER: I am new to sklearn and open-source projects in general, so sorry if I don't comply with the standards of this projects.
Problem:
So, say you have a learner which gets as input also the column type and processes accordingly (e.g. it converts categorical to numbers but leave floats as is). How would you put such a learner in a Pipeline? You could have do this with the buildin feature-extraction and preprocessing of scikit-learn, but say you have an external code which does that, or say you want the imputation to be a function of column type, and you would like to do forward-feature-selection without pre-imputing (the imputation is part of the pipeline).

Suggestion:
I believe we should support meta-data across the columns and rows of the feature matrix to allow passing information on features (e.g. type of feature) and samples (e.g. weight of instance).
Maybe this is easily implementable through multiindex in pandas (http://pandas.pydata.org/pandas-docs/stable/advanced.html)

What do you say?
Maybe I am missing something and there is a way to do that with the current scikit-learn design?

Thanks,
HS

user: chanansh
issue_4194: state: open
title: kneighbors_graph param check ineffective
body: In `sklearn.neighbors.graph`, the `_check_params` function is called by both of the graph methods in the module. The idea seems to be to ensure that when the neighbors are precomputed that the parameters passed into say, `kneighbors_graph` match what was used in the computation of the `BallTree`. However, if the `BallTree` itself was from a prior fit, the check misses that fact. Demonstration:

``` python
data = np.random.rand(100).reshape(10,10)
n_neighbors = 2
neigh_minkowski = neighbors.NearestNeighbors(n_neighbors, metric='minkowski')
neigh_jaccard = neighbors.NearestNeighbors(n_neighbors, metric='jaccard')
neigh_jaccard.fit(data)

# "Copies" _fit_X, _tree, _fit_method
# But it does not copy p, metric, effective_metric, effective_metric_params, etc.
neigh_minkowski.fit(neigh_jaccard)
assert(neigh_minkowski.metric == 'minkowski') # misleading, but explainable

# Succeeds
neighbors.kneighbors_graph(neigh_minkowski, n_neighbors, mode='distance', metric='minkowski')

# Fails: ValueError: Got jaccard for metric, while the estimator has minkowski for the same parameter.
neighbors.kneighbors_graph(neigh_minkowski, n_neighbors, mode='distance', metric='jaccard')
```

I'm not sure any of this matters since `kneighbors_graph` (and the things it calls) doesn't do anything with the name of the metric if the BallTree has been precomputed. But it's certainly the case the `_check_params()` is not having its intended effect. So does this matter? Should `NearestNeighbor._fit()` be modified so that it copies all relevant data over when X is "precomputed"?

This is a bit contrived, but I ran across it while trying to modify `Isomap` to handle non-Euclidean metrics. The issue is that Isomap initializes a `NearestNeighbors` instance with metric equal to 'minkowski' and `p=2`. So if you call `fit` with a precomputed `NearestNeighbors` using a different metric, then this "fact" is not transferred to the initialized instance. This can all be avoided if the API to Isomap was changed, but this particular point of `_check_params()` not always doing an effective check still holds.

user: chebee7i
issue_4170: state: open
title: Contamination and threshold should be part of covariance.OutlierDetectionMixin.predict
body: The settings of contamination and threshold are currently possible only at the **init** of the covariance.EllipticEnvelope and covariance.OutlierDetectionMixin:

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/covariance/outlier_detection.py#L37
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/covariance/outlier_detection.py#L168

These may be the precomputed defaults. However, it would be much more useful if the predict method accepted the contamination/threshold parameters, too. These parameters are used only to make the output of the decision_function binary, i.e. turn values above the threshold into -1 and the rest into 1.

Thanks for improving these!

user: otakar-smrz
labels: Needs Decision - Include Feature: Requires decision regarding including feature
issue_4143: state: open
title: Allow for Transformers on y
body: Following up on #3113 and #3112, what about arbitrary transforms to the `y` values? Those issues dealt primarily with "label transforms"  but I would like to use transformers to mean or range center the `y` values as well.

Ideally I would have some transform that can be applied to the y values before fitting and then applied in the inverse to the predicted `y` values coming out of `predict`.

Ideally this Transformer could be added to a pipeline.

Currently the signature for `transform` for [`StandardScaler`](https://github.com/scikit-learn/scikit-learn/blob/a413f875f1a7c9528290e01e1c030c336b9f32e4/sklearn/preprocessing/data.py#L338) allows for transforming `y` but as pointed out in the linked issues, not all transform methods have a signature allowing for a `y` to be passed in. 

Further even for `StandardScaler` there is an inconsistency with the [`inverse_transform`](https://github.com/scikit-learn/scikit-learn/blob/a413f875f1a7c9528290e01e1c030c336b9f32e4/sklearn/preprocessing/data.py#L366) NOT taking `y`.

user: cancan101
issue_4134: state: open
title: Test coverage and instability in GraphLasso
body: It looks to me as if `GraphLasso` doesn't have a very good test coverage, for example `mode != 'cd'` is not tested.
Also, this warning:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/covariance/graph_lasso_.py#L238

can cause an error if `max_iter==0` which happens on random data (run any of the common tests).

user: amueller
issue_4087: state: open
title: [MRG] support sample_weight in silhouette_score
body: I sought `sample_weight` in `silhouette_score`, to account for multiple points that are merged into one when calculating average distances. Hacking it into the current implementation resulted in a very slow solution. Thus this PR also rewrites the implementation, yielding something that's a bit slower than the solution at master, but supports `sample_weight`.

I've also added tests for correctness which I haven't otherwise found in the code.

user: jnothman
issue_4062: state: open
title: KernelDensity and GMM interfaces are unnecessarily confusing
body: To evaluate the density of `data` at `x`, the current interface is `density = np.exp(kde.score_samples(data))`

(see in action [here](http://nbviewer.ipython.org/gist/jakevdp/0b3e332015ec2de7b973))

This seems really silly and unnecessary for such a common operation.

How would people feel about adding a `density` method to `KernelDensity` and to `GMM` to make this easier for users to figure out?

user: jakevdp
labels: Needs Decision - Close: Requires decision for closing
issue_4040: state: open
title: [ENH]: add example for cluster criteria
body: Unfortunately, there does not seem to be any example on how clustering results (e.g., k-means clustering) can be weighed and evaluated using cluster validity criteria. How about adding a small new example that shows how to realize cluster validity assessment by
- variation of information
- silhouette coefficient
- intra/extra cluster distance ratio
- Jaccard index, as a variant of dice coefficient

What do you think @ogrisel , @GaelVaroquaux and @dengemann ?

user: banilo
issue_3998: state: open
title: Support sample weight in clusterers
body: Currently no clusterers (or clustering metrics) support weighted dataset (although support for DBSCAN is proposed in #3994).

Weighting can be a compact way of representing repeated samples, and may affect cluster means and
variance, average link between clusters, etc.

Ideally BIRCH's global clustering stage should be provided a weighted dataset, and is current use of unweighted representatives may make its parametrisation more brittle.

This could be subject to an invariance test along the lines of:

```
sample_weight = np.random.randint(0, 10, size=X.shape[0])
weighted_y = clusterer.fit_predict(X, sample_weight=sample_weight)
repeated_y = clusterer.fit_predict(np.repeat(X, sample_weight))
assert_equal(adjusted_rand_score(np.repeat(weighted_y, sample_weight), repeated_y)
# NB: this is only a useful sufficient test if weighted_y differs from clusterer.fit_predict(X)
```

(There is also a minor question of whether `sample_weight` should be universally accepted by `ClusterMixin` or whether `WeightedClusterMixin` should be created, etc.)

Sample weight support for clusterers:
- [ ] Affinity propagation (I don't know this well enough to know the applicability)
- [ ] BIRCH
- [x] DBSCAN
- [ ] Hierarchical -> Ward link
- [x] Hierarchical -> Complete link (N/A, as far as I can tell)
- [ ] Hierarchical -> Average link
- [x] K Means
- [x] Minibatch K Means
- [ ] Mean shift
- [ ] Spectral

user: jnothman
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_3907: state: open
title: [WIP] Adding tests for estimators implementing `partial_fit` and a few other related fixes / enhancements
body: Fixes #3896 

**New `partial_fit` tests:**
The below tests are based off @arjoly's [suggestion](https://github.com/scikit-learn/scikit-learn/issues/3896#issuecomment-64862823).
- [x] 1. General tests
  - Assert that `partial_fit` returns `self`. - Thanks @jnothman for [the suggestion.](https://github.com/scikit-learn/scikit-learn/issues/3896#issue-50330161)
  - Clone test... `assert clone(est).partial_fit == est.partial_fit`
- [x] 2. Check that an error is raised if the number of features changes.
- [x] 3. Estimator reset test
  - Check that doing `fit` after a set of `fit` / `partial_fit` restarts the estimator.
  - Assert that `partial_fit` and then `fit` with a different number of features works without raising any Exceptions... - Thanks @amueller for [the suggestion](https://github.com/scikit-learn/scikit-learn/pull/3907#issuecomment-68411916)
- [x] 4. Check if `partial_fit` does not overwrite the previous model.
  - Based on @jnothman [suggestion](https://github.com/scikit-learn/scikit-learn/pull/3907#issuecomment-68478630)
- [x] 5. Check that classifier handles correctly the classes argument in the `partial_fit`.
  - a. Check if mismatch between `classes` argument and `np.unique(y_i)` raises `ValueError`.
  - b. Check that error is raised if `classes` is not specified during first `partial_fit` call.
  - c. Check error is not raised if `classes` is not specified during subsequent calls.
- [x] Helper functions
  - `_validate_y_against_classes` - Check label mismatch between `y` and `classes` arg
  - `assert_same_model(X, est1, est2)`
  - `assert_not_same_model(X, est1, est2)`
    - Use `predict`, `transform`, `decision_function` and `predict_proba` to check equality of models.
  - In `sklearn.utils.estimator_checks` `_partial_fit` and `_fit` to use the appropriate parameters.
  - `assert_attributes_equal(est1, est2)`
  - `assert_attributes_not_equal(est1, est2)`
  - `assert_array_not_equal`

**Refs**
Also see #406 - This PR fixes 1st under not so easy for pfit-able estimators, via 3a ( of this PR )

<!-- Reviewable:start -->

---

This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/scikit-learn/scikit-learn/3907)

<!-- Reviewable:end -->

user: raghavrv
issue_3896: state: open
title: Invariance testing for partial_fit
body: Much of the common functionality across estimators is tested within `sklearn.tests.test_common`. As far as I can tell, there are no tests of what `partial_fit` should do in general, such as returning `self`. (I'm not sure what else is general to all estimators supporting `partial_fit`.) Test(s) should be added.

user: jnothman
labels: Hard: Hard level of difficulty
module:test-suite: everything related to our tests
issue_3855: state: open
title: Resampler estimators that change the sample size in fitting
body: Some data transformations -- including over/under-sampling (#1454), outlier removal, instance reduction, and other forms of dataset compression, like that used in BIRCH (#3802) -- entail altering a dataset at training time, but leaving it unaltered at prediction time. (In some cases, such as outlier removal, it makes sense to reapply a fitted model to new data, while in others model reuse after fitting seems less applicable. )

As noted [elsewhere](https://github.com/scikit-learn/scikit-learn/pull/1454#issuecomment-11313359), transformers that change the number of samples are not currently supported, certainly in the context of `Pipeline`s where a transformation is applied both at `fit` and `predict` time (although a hack might abuse `fit_transform` to make this not so). `Pipeline`s of `Transformer`s also would not cope with changes in the sample size at fit time for supervised problems because `Transformer`s do not return a modified `y`, only `X`.

To handle this class of problems, I propose introducing a new category of estimator, called a `Resampler`. It must define at least a `fit_resample` method, which `Pipeline` will call at `fit` time, passing the data unchanged at other times. (For this reason, a `Resampler` cannot also be a `Transformer`, or else we need to define their precedence.)

For many models, `fit_resample` needs only return `sample_weight`. For sample compression approaches (e.g. that in BIRCH), this is not sufficient as the representative centroids are modified from the input samples. Hence I think `fit_resample` should return altered data directly, in the form of a dict with keys `X`, `y`, `sample_weight` as required. (It still might be appropriate for many `Resampler`s to only modify `sample_weight`; if necessary, another `Resampler` can be chained that realises the weights as replicated or deleted entries in `X` and `y`.)

user: jnothman
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_3774: state: open
title: Ensemble models (and maybe others?) don't check for negative sample_weight
body: When sample weights are negative, the probabilities can come out negative as well:

```
>>> rng = np.random.RandomState(10)
>>> X = rng.randn(10, 4)
>>> y = rng.randint(0, 2, 10)
>>> sample_weight = rng.randn(10)
>>> clf = RandomForestClassifier().fit(X, y, sample_weight)
>>> clf.predict_proba(X)
array([[ 0.56133774,  0.43866226],
       [ 1.03235924, -0.03235924],
       [ 1.03235924, -0.03235924],
       [ 1.03235924, -0.03235924],
       [ 1.03235924, -0.03235924],
       [ 1.03235924, -0.03235924],
       [ 0.98071868,  0.01928132],
       [ 0.56133774,  0.43866226],
       [ 1.03235924, -0.03235924],
       [ 1.03235924, -0.03235924]])
```

user: larsmans
issue_3739: state: open
title: [MRG] K-SVD
body: Implemented a K-SVD algorithm as a free function.

Notes/questions:
- I haven't done any optimizations yet, but I left TODO-notes in places where it can be done. I am going to profile the code later and see where/if the optimizations are necessary.
- The algorithm currently uses OMP as a sparse coder, and it is possible to make it customizable. However this will make the function interface more complicated (as different sparse coders have different parameters). Should I do it, or should I leave it like this?
- There are now no unit-tests for the implementation, but I am a bit confused on how to create them for such functions. Should I just check the API and simple functions like `_worst_represented_example`?

user: alexeyum
issue_3737: state: open
title: DBScan Clustering of string data
body: So, I want to use the DBScan clustering algorithm from scikit learn, on a dataset of string charcaters. So, I used metric=Levenshtein.distance and my data is given in a numpy array. But, why it wants to convert them to float?

I am using the following packages:

```
CPython 2.7.3
IPython 2.0.0

numpy 1.9.0
scipy 0.14.0
scikit-learn 0.15.2
matplotlib 1.4.0
pyprind 2.5.0
```

and my data shape and dbscan object is defined as below:

```
d_arr.shape
(30079, 1)

dbs = sklearn.cluster.DBSCAN(eps=4, min_samples=2, metric=Levenshtein.distance)
```

```
dbs.fit(d_arr)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-33-a7573a2d5bba> in <module>()
----> 1 dbs.fit(d_arr)

/usr/local/lib/python2.7/dist-packages/sklearn/cluster/dbscan_.pyc in fit(self, X)
    246             Overwrite keywords from __init__.
    247         """
--> 248         clust = dbscan(X, **self.get_params())
    249         self.core_sample_indices_, self.labels_ = clust
    250         self.components_ = X[self.core_sample_indices_].copy()

/usr/local/lib/python2.7/dist-packages/sklearn/cluster/dbscan_.pyc in dbscan(X, eps, min_samples, metric, algorithm, leaf_size, p, random_state)
     99                                            leaf_size=leaf_size,
    100                                            metric=metric, p=p)
--> 101         neighbors_model.fit(X)
    102 
    103     # Calculate neighborhood for all samples. This leaves the original point

/usr/local/lib/python2.7/dist-packages/sklearn/neighbors/base.pyc in fit(self, X, y)
    659             Training data. If array or matrix, shape = [n_samples, n_features]
    660         """
--> 661         return self._fit(X)

/usr/local/lib/python2.7/dist-packages/sklearn/neighbors/base.pyc in _fit(self, X)
    232             self._tree = BallTree(X, self.leaf_size,
    233                                   metric=self.effective_metric_,
--> 234                                   **self.effective_metric_params_)
    235         elif self._fit_method == 'kd_tree':
    236             self._tree = KDTree(X, self.leaf_size,

/usr/local/lib/python2.7/dist-packages/sklearn/neighbors/ball_tree.so in sklearn.neighbors.ball_tree.BinaryTree.__init__ (sklearn/neighbors/ball_tree.c:7983)()

/usr/local/lib/python2.7/dist-packages/numpy/core/numeric.pyc in asarray(a, dtype, order)
    460 
    461     """
--> 462     return array(a, dtype, copy=False, order=order)
    463 
    464 def asanyarray(a, dtype=None, order=None):

ValueError: could not convert string to float: TACGTAGGGGGCAA
```

user: vmirly
issue_3652: state: open
title: efficient grid search for random forests
body: To get the best performance off Random Forests, it is necessary to tune parameters like `max_depth`, `min_samples_split` and `min_samples_leaf`. If we wrap a RF in GridSearchCV, trees are built from scratch every time. However, for depth first tree induction, deeper trees share the same base as shallower trees. An idea to speed up grid search is thus to not rebuild trees from scratch every time. Here's an example for `max_depth=[10, 9, .., 1]`.
1. Set `max_depth=10`
2. Build `n_estimators` fully developed trees
3. Prune trees to have a maximum depth of `max_depth`
4. Create a RF for this `max_depth` and evaluate it using the current train/test split
5. Decrease `max_depth` and go to step 3

Such an algorithm could be wrapped in `RandomForestClassifierCV` / `RandomForestRegressorCV`  classes.

user: mblondel
issue_3547: state: open
title: ENH: Add parameter pmax that is present in glmnet
body: Glmnet has a parameter called `pmax` that limits the non zero coefficients, which would help a lot in the sparse case with l1 norm. I was wondering if we needed to add this parameter to ElasticNet in sklearn.

user: MechCoder
issue_3529: state: open
title: ENH: Add a default label to LabelPropagation algorithms (fixes #3218)
body: This PR addresses the issues raised in #3218. There are two changes that might affect users:
1. A new kwarg is added to `LabelPropagation`/`LabelSpreading` initializers, which lets the user set default labels for instances that aren't labeled by the `fit()` call. This defaults to `None`, which means that the presence of any unlabeled instances will trigger an exception. If not `None`, the `transduction_` array will be given default values for unlabeled instances.
2. The termination condition in `fit()` had an off-by-one error, where setting `max_iter=1` was iterating zero times. This has been fixed.

I've also supplied a test case that exercises this new behavior.

user: perimosocordiae
issue_3482: state: open
title: Generalized additive models (GAMs)?
body: Hello there,
Thanks for making this fantastic library. I use it every day in my bioinformatics research. We're developing a toolkit for single-cell RNA-seq analysis (http://github.com/yeolab/flotilla) and want to add all current state-of-the-art analyses. Unfortunately, most of these are in R. I can reimplemement some of them, but they rely on certain R packages, in particular [VGAM](http://cran.r-project.org/web/packages/VGAM/index.html), aka Vector Generalized Linear and Additive Models. I've found a few mentions of GAMs here:
- https://github.com/scikit-learn/scikit-learn/wiki/A-list-of-topics-for-a-google-summer-of-code-(gsoc)-2012
- A subtype of GAMs, Multiple additive regression (MARS) discussion: https://github.com/scikit-learn/scikit-learn/issues/845
  - MARS pull request https://github.com/scikit-learn/scikit-learn/pull/2285

Has there been any update on creating these libraries?

user: olgabot
issue_3466: state: open
title: Explanation of nu parameter in One-class SVM
body: In the latest documentation for Outlier Detection (http://scikit-learn.org/stable/modules/outlier_detection.html), it mentioned an important distinction between novelty detection and outlier detection is that: In novelty detection, "the training data is not polluted by outliers", and in outlier detection, "the training data contains outliers". And in the example, One-class SVM is used to demonstrate novelty detection.

However, in One-class SVM, it is still possible to accept outliers in the training data. Particularly, the parameter nu is used to tune upper bound on the fraction of outliers in the training dataset (as explained in Proposition 4 of the original paper - Estimating the support of a high-dimensional distribution, by B Schölkopf et al.).

So I think that the distinction between outlier detection and novelty detection is not well-illustrated in the current documentation (by the use of One-class SVM). And in fact, I would think that we should not differentiate between the two cases.

Besides, the current explanation of nu parameter(i.e., "The  \nu parameter, also known as the margin of the One-Class SVM, corresponds to the probability of finding a new, but regular, observation outside the frontier.") should be re-written based on the explanation from the original paper to make things clearer.

user: pvnguyen
issue_3455: state: open
title: Allow to choose the out-of-bag scoring metric
body: Estimators in the forest module (random forest and extra trees) and in the bagging module allows to compute the out-of-bag estimates of the performance of the forest. 

A nice things to add would to allow the choice of the scoring function using the scorer interface. The `oob_score` parameter would be equal would be the string corresponding to the appropriate scorer. 

Thus, you would have 

```
oob_score : bool or string, (default=False)
        Whether to use out-of-bag samples to estimate
        the generalization error. The oob scoring function could be chosen
        by passing  a string (see model evaluation documentation) or
        a scorer callable object / function with signature
        ``scorer(estimator, X, y)``.

```

user: arjoly
labels: Needs Decision: Requires decision
issue_3453: state: open
title: Add multioutput-multiclass support to metrics
body: Some estimators such as trees support multi-output multiclass. However, there isn't any metric yet to assess those tasks. Here a list of metrics that could be easily extended to handle this format:
1. accuracy_score or subset accuracy and zero_one_loss or subset zero-one loss
2. hamming_loss

Ideally, it would be one pull request for point (1) and (2).

user: arjoly
issue_3449: state: open
title: Add multi-output support to the bagging module
body: This would nice to add multi-output support to bagging as in the `sklearn.ensemble.forest` module. Some code could be refactored.

user: arjoly
issue_3383: state: open
title: [WIP] Ridge path
body: Adressing https://github.com/scikit-learn/scikit-learn/issues/3373

A first implementation of a ridge path using `'eigen'` solver. This solver is now available in `ridge_regression`, which calls upon a reduced functionality of `ridge_path`.

Next steps:
- The same path approach can be straightforwardly used with SVD as well: Write svd path and replace in ridge_regression
- Attack `RidgeCV`: Replace the `GridSearchCV` by `ridge_path` if `eigen` or `svd` are selected.
- Incorporate `ridge_path_looe` from the gist https://gist.github.com/eickenberg/e9eb106f12e40d017fd0
- Adjust `RidgeCV` to use the looe functionalities

user: eickenberg
issue_3379: state: open
title: [ENH]: add InfomaxICA object and function
body: We recently added a pure Numpy implementation of the ICA infomax algorithm to [mne-python](https://github.com/mne-tools/mne-python).

https://github.com/mne-tools/mne-python/blob/master/mne/preprocessing/infomax_.py
https://github.com/mne-tools/mne-python/blob/master/mne/preprocessing/tests/test_infomax.py

It should not be too difficult to include it in sklearn since we already have tests that are adapted from the FastICA tests.

If people are interested we could start discussing the API.

cc @GaelVaroquaux @ogrisel @agramfort 

user: dengemann
issue_3373: state: open
title: [ENH] Ridge Path and general refactoring
body: Hi everybody,

during the sprint (July 14th 2014 to 20th), I am hoping among other things to iron out some inconsistencies in `ridge.py` and add a nice-to-have `ridge_path` (as proposed in https://github.com/scikit-learn/scikit-learn/issues/582) which conveniently provides coefficients or predictions necessary for the CV parameter selection algorithms.

The `RidgeClassifierCV` classifier also needs some care, since its inner cross-validation loop is invariably leave-one-out MSE. Other inner loop scoring should be possible (should be easy with `ridge_path`) or the classifier renamed (see https://github.com/scikit-learn/scikit-learn/pull/3302).

My idea is to first write `ridge_path`, and then gently move all functionality that could use it over.

A gist for `ridge_path` is here https://gist.github.com/eickenberg/e9eb106f12e40d017fd0

All of the functions are multi-penalty, multi-target. I would like to systematically broadcast penalties according to rules presented in https://github.com/scikit-learn/scikit-learn/pull/2068

I am still not completely sure where to place certain decision logic (i.e. solvers, etc). `ridge_path` seem like something that could be in the public API, and should thus probably contain decision logic and a `solver=` kwarg.

Also, I hear talk of a potential `KernelMixin`. I don't know what it should do, but I coded a kernel function as one taking two matrices X, and Y, and, treating lines as samples, returns a similarity matrix of all the samples.

Take this as a roadmap and request for comments -- very happy about any input!

@mblondel @ogrisel @fabianp @agramfort @GaelVaroquaux @bthirion

user: eickenberg
issue_3332: state: open
title: Making KernelDensity a parameter of MeanShift
body: It would be really great if one could choose a sklearn.neighbors.KernelDensity object to be used by the sklearn.cluster.MeanShift class. For my personal use case, I need to run MeanShift with the haversine distance metric on a gaussian kernel.
What do you think? 

user: cbarokk
issue_3271: state: open
title: prediction variance in bagging-based regressors
body: It would be nice if we could do:

``` python
rf = RandomForestRegressor()
rf.fit(X, y)
y_mean, y_var = rf.predict(X, ret_variance=True)
```

Same for BaggingRegressor.

user: mblondel
issue_3270: state: open
title: KernelDensity docstring
body: I had troubles understanding how parameters related to KD-tree and Ball tree affect DensityEstimation. 

In my understanding, KDE uses the average evaluation of kernels centered on _every_ training point so I didn't quite understand how KD-tree and Ball-tree are useful (I would understand if it used, say, the average of the k nearest neighbors only).

The docstring says that we can specify tolerance options `rtol` and `atol`. With respect to what stopping criterion are these constants used?

Regarding the breadth_first option, the docstring says "use a breadth-first approach to the problem". I didn't understand what "problem" it refers to.

What is the practical impact of of the tree related parameters? Do they only affect speed or can they also affect quality of estimation?

Sorry for the naive questions but it would help my understanding if we could add a couple of words to clarify the docstring.

BTW, thanks @jakevdp for this great module!

user: mblondel
issue_3218: state: open
title: ENH: support unlabeled output in label propagation
body: Currently, when the [label propagation algorithm](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/semi_supervised/label_propagation.py#L244) hits `max_iter`, it's possible for some rows of `self.label_distributions_` to be entirely zero.

This makes sense, because labels haven't yet propagated to those samples. Unfortunately, during the normalization step this turns those label distributions into NaN values, which results in the first label being assigned (`np.argmax(nan_array) == 0`). This also issues a RuntimeWarning about the divide by zero.

 I believe it would be better to give the user an option in this case, perhaps to:
- set these labels to -1 to indicate no label (similar to the input format)
- set these labels to some user-supplied value (perhaps defaulting to 0 to preserve behavior)

Currently, only the warning gives any indication that some labels may not have propagated fully. This is awkward to catch in user code, and there's no easy way to determine which samples failed to be labeled.

If this makes sense, I'll be happy to provide a PR implementing the change.

user: perimosocordiae
issue_3130: state: open
title: winsorization/trimming in preprocessing
body: A useful preprocessing step, especially when doing prototyping, is to winsorize, trim, or clip the data to some hard limit. I've searched through the repo and mailing list but don't see anything like this currently implemented (or proposed). I'd be willing to submit a patch if there's support for this; it would provide both a standalone method and a class that follows the Transformer API.

user: tyarkoni
issue_3126: state: open
title: Updated K-means clustering for Nystroem
body: Because I wanted to try K-means clustering as the basis for Nystroem approximation and it appeared as though pull request #2591 might be stalled I created a slightly modified version. I also tried to address @amueller comment about the effectiveness of the method by including it in the plot_kernel_approximation example and @dougalsutherland comment concerning the possible singularity of the sub-sampled kernel matrix using the same approach as scipy does in pinv2.

Since it is my first commit to the project (hopefully the first of many) any feedback or suggestions you have would be appreciated.

user: nateyoder
labels: Needs Benchmarks: A tag for the issues and PRs which require some benchmarks
issue_3075: state: open
title: Local Regression is Missing
body: Is local linear regression in scikit? Are you planning to implement that in the future? If not, are contributions welcome in this area?

user: aboSamoor
issue_3031: state: open
title: Feature request: Stability of feature subset selection
body: One possible criterion to judge the outcome of a feature subset selection is how much the selection depends on the training set. 

The Stability estimator runs the feature selection on different subsamples. It then computes the cumulative similarity of the selected feature subsets. Two possible measures for this similarity are the jaccard similarity or kunchevas index.

See: 
Saeys, Yvan, Thomas Abeel, and Yves Van de Peer. "Robust feature selection using ensemble feature selection techniques." Machine learning and knowledge discovery in databases. Springer Berlin Heidelberg, 2008. 313-325.

Kunchevas index:
Kuncheva, Ludmila I. "A stability index for feature selection." Artificial Intelligence and Applications. 2007.

Edit: I'm sorry, but it forgot to set a label for this issue. Apparently I cannot relabel this once posted.

user: ghost
issue_2855: state: open
title: GBRT monitor held out set
body: I was toying with the new monitor callback and ran into an issue. I'm not sure if I'm using it incorrectly or if it's a bug. So I have something like the following.

```
def monitor(i, est, locals):
    print('test = %f' % loss(y_test, est.decision_function(X_test)))
reg = GBRT()
reg.fit(X_train, y_train, monitor=monitor)
```

Which results in
`AttributeError: 'NoneType' object has no attribute 'tree_'`
due to `self.estimators_` containing `None` elements.

```
array([[ DecisionTreeRegressor(compute_importances=None,
           criterion=<sklearn.tree._tree.FriedmanMSE object at 0x5208030>,
           max_depth=6, max_features=None, max_leaf_nodes=None,
           min_density=None, min_samples_leaf=1, min_samples_split=2,
           random_state=<mtrand.RandomState object at 0x7f52ec7d45b8>,
           splitter=<sklearn.tree._tree.PresortBestSplitter object at 0x520b050>)],
       [None],
       [None],
       [None],
...
```

user: jwkvam
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_2807: state: open
title: Add support for SVDD (one-class SVM)
body: Support Vector Data Description (SVDD) could be a nice enhancement to OneClassSVM implementation.

A technical implementation is described in this paper:
http://www.csie.ntu.edu.tw/~cjlin/papers/svdd.pdf

Source code compatible with libsvm is available here:
http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/#libsvm_for_svdd_and_finding_the_smallest_sphere_containing_all_data

user: don-vip
labels: Needs Decision: Requires decision
issue_2735: state: open
title: Numerical stability of symmetric decorrelation in FastICA
body: I'm trying to use `fastica` with a custom contrast function that ought to maximise the skewness of the ICs:

```
skew(x, fun_args=None):
    return x*x, (2*x).mean(axis=-1)
```

I'm basically aiming to replicate the functionality of [`fastica.m`](https://abacus.abo.fi/ro.nsf/ByCat_RO/34E42AC34DB32F7CC2257B790021B5D3/$FILE/fastica.m) with `'skew'` chosen as the nonlinearity.

However, I'm running into an error in the symmetric decorrelation step:

```
/home/alistair/.venvs/slices/lib/python2.7/site-packages/sklearn/decomposition/fastica_.py in fastica(X, n_components, algorithm, whiten, fun, fun_args, max_iter, tol, w_init, random_state, return_X_mean, compute_sources)
    318 
    319     if algorithm == 'parallel':
--> 320         W = _ica_par(X1, **kwargs)
    321     elif algorithm == 'deflation':
    322         W = _ica_def(X1, **kwargs)

/home/alistair/.venvs/slices/lib/python2.7/site-packages/sklearn/decomposition/fastica_.py in _ica_par(***failed resolving arguments***)
    102         gwtx, g_wtx = g(np.dot(W, X), fun_args)
    103         W1 = _sym_decorrelation(np.dot(gwtx, X.T) / p_
--> 104                                 - g_wtx[:, np.newaxis] * W)
    105         del gwtx, g_wtx
    106         lim = max(abs(abs(np.diag(np.dot(W1, W.T))) - 1))

/home/alistair/.venvs/slices/lib/python2.7/site-packages/sklearn/decomposition/fastica_.py in _sym_decorrelation(W)
     45     i.e. W <- (W * W.T) ^{-1/2} * W
     46     """
---> 47     s, u = linalg.eigh(np.dot(W, W.T))
     48     # u (resp. s) contains the eigenvectors (resp. square roots of
     49     # the eigenvalues) of W * W.T

/home/alistair/.venvs/slices/lib/python2.7/site-packages/scipy/linalg/decomp.pyc in eigh(a, b, lower, eigvals_only, overwrite_a, overwrite_b, turbo, eigvals, type, check_finite)
    290     """
    291     if check_finite:
--> 292         a1 = asarray_chkfinite(a)
    293     else:
    294         a1 = asarray(a)

/home/alistair/.venvs/slices/lib/python2.7/site-packages/numpy/lib/function_base.pyc in asarray_chkfinite(a, dtype, order)
    603     if a.dtype.char in typecodes['AllFloat'] and not np.isfinite(a).all():
    604         raise ValueError(
--> 605             "array must not contain infs or NaNs")
    606     return a
    607 

ValueError: array must not contain infs or NaNs
```

The problem seems to be due to the fact that the eigenvalue decomposition here:

```
def _sym_decorrelation(W):
    """ Symmetric decorrelation
    i.e. W <- (W * W.T) ^{-1/2} * W
    """
    s, u = linalg.eigh(np.dot(W, W.T))
    # u (resp. s) contains the eigenvectors (resp. square roots of
    # the eigenvalues) of W * W.T
    return np.dot(np.dot(u * (1. / np.sqrt(s)), u.T), W)
```

sometimes returns negative eigenvalues, which yield NaNs in the square root.

I tried using a more numerically stable method for orthonormalizing `W` using the SVD:

```
def _sym_decorrelation(W):
    """ Symmetric decorrelation via SVD
    i.e. W == U*S*V.T
         W <- U*V.T
    # """
    U, _, Vt = linalg.svd(W, full_matrices=False, overwrite_a=True, 
                                 check_finite=False)
    return U.dot(Vt)
```

This is incurs a slightly greater computational cost, but this can be largely mitigated by overwriting `W` in place and disabling checks for NaNs and Infs (`svd` will simply raise a `LinAlgError` if it encounters these anyway). In any case, evaluating the contrast function seems to be more of a bottleneck than the decorrelation step.

With this more stable decorrelation function, `fastica` converges successfully and I get sensible-looking ICs.

I can't exactly call this a bug, since I only seem to hit this error using my custom contrast function, but I thought I'd point out that there are more numerically stable methods for the decorrelation step.

user: alimuldal
issue_2709: state: open
title: Add `best_score_` attribute to all CV models
body: GridSearchCV has a `best_score_` attribute that is missing from RidgeCV, LassoCV, and other CV based objects. Ideally, these would all have a consistent API since they're all performing essentially the same task: from a given set of parameters, search for the optimal one according to the cross validation score.

Other attributes that may be beneficial to add:
- `best_estimator_` : This may be useful by returning a fit object of the non-CV version of the class (e.g. RidgeCV --> Ridge)
- `best_params_` : This would be helpful, especially for ElasticNetCV which has multiple parameters set (`alpha_` and `l1_ratio_`).
- `grid_scores_` : This is mostly available for LassoCV, but will take more work to make available for RidgeCV.

If no one objects, I'll start working on this. Ideas for how to improve or other attributes to add are welcome.

user: eshilts
issue_2648: state: open
title: Add additional bandwidth settings to KernelDensity
body: Hi,

The new KernelDensity class is really great! There are a few things I might like to see added in future releases:
1. Per dimension bandwidth control. I'd like to be able to pass in an array of bandwidths and apply different smoothing in each dimension.
2. Adaptive smoothing. I'd like to be able to apply variable smoothing such that the bandwidth is chosen equal to the kth nearest neighbor distance at each sample point.

Does anyone have an idea of how easy/possible these features might be to implement? I had a look through the code but don't know enough about tree-based evaluation to tell if it could support these functions.

Thanks!

user: alexsparrow
labels: Needs Decision - Close: Requires decision for closing
issue_2639: state: open
title: Use succinct tries for vocabulary storage
body: Hey,

Python dictionary is quite wasteful for string data, so `CountVectorizer`, `TfidfVectorizer` and `DictVectorizer` all could take much less memory with an another data structure for `vocabulary_`. 

Here is a quick prototype that persists `vocabulary_` of `CountVectorizer` and `TfidfVectorizer` in [MARISA-Trie](https://code.google.com/p/marisa-trie/) via Python [wrapper](https://github.com/kmike/marisa-trie): https://gist.github.com/kmike/7814472.

I used [this](https://gist.github.com/kmike/7813450) script to measure memory and speed of `fit` and `dump`, and [this](https://gist.github.com/kmike/7815149) script to measure parameters of `load` and `transform`.

The results are quite cool: for '20 newsgroups' data (fit on training subset, transform on test subset) memory consumption of different vectorizers after loading is the following:
-  `CountVectorizer()`: 94MB;
-  `CountVectorizer(ngram_range=(1,2)`: 666MB;
-  `MarisaCountVectorizer()`: 1.2MB;
-  `MarisaCountVectorizer(ngram_range=(1,2))`: 13.3MB;

So using a good succinct Trie implementation gives us 50x-80x reduction of memory usage; it also makes serializing and deserializing almost instant. And such vectorizers don't have HashingVectorizer limitations.

The downside is that `fit` method is 2x-3x slower and requires slightly more memory (but I think it could be changed to require less memory than CountVectorizer.fit), and `transform` is about 2x slower. Full output: https://gist.github.com/kmike/7815156

What do you think about adding something similar to scikit-learn? 

user: kmike
issue_2606: state: open
title: Add support for: ML-kNN
body: Add support for the Multi-Label kNN algorithm as described here in this paper http://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/pr07.pdf. A brief description of the algorithm from the above paper:

> "As its name implied, Ml-knn is derived from the popular k-Nearest Neighbor (kNN) algorithm [1]. Firstly, for each test instance, its k nearest neighbors in the training set are identi¯ed. Then, according to statistical information gained from the label sets of these neighboring instances, i.e. the number of neighboring instances belonging to each possible class, maximum a posteriori (MAP) principle is utilized to determine the label set for the test instance."

Supporting sparse matrices should be a key requirement - a lot of multi label tasks are for text categorization and these are usually represented as sparse matrices.

ML-kNN is actually already implemented in this library (http://orange.biolab.si/docs/latest/reference/rst/Orange.multilabel/#ml-knn-learner) but it would be good to bring it under the scikit-learn framework as well.

user: mchangun
labels: Low Priority: Low priority issues and pull requests
issue_2542: state: open
title: port fast/better ARD regression from MNE-Python
body: https://github.com/mne-tools/mne-python/blob/master/mne/inverse_sparse/_gamma_map.py#L17

to replace "historical" ARDRegression model.

this issue is just a reminder for me

cc @mluessi

user: agramfort
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_2484: state: open
title: Bi-/coclustering API prevents scalability
body: The biclustering and coclustering estimators promise to store boolean arrays `rows_` and `columns_`, of size `n_clusters * n_samples` and `n_clusters * n_features`, respectively, and convert these to indices only on demand. For use with large, sparse matrices and large numbers of clusters, these should be arrays of indices rather than boolean masks.

user: larsmans
labels: Moderate: Anything that requires some knowledge of conventions and best practices
Needs Decision: Requires decision
issue_2451: state: open
title: Multi-label and multi-output multi-class decision functions and predict proba aren't consistent
body: The `decision_function` and `predict_proba` of a multi-label classifier (e.g. `OneVsRestClassifier`) is a 2d arrays where each column correspond to a label and each row correspond to a sample. (added in 0.14?)

The `decision_function` and `predict_proba` of multi-output multi-class classifier (e.g. `RandomForestClassifier`) is a list of length equal to the number of output with a multi-class decision_function or predict_proba output (a 2d array where each row corresponds to the samples and where each columns correspond to a class).

So this means that multi-output problem with only binary class output is a multi-label task, but isn't consistent with the multi-label format...

This is problematic if you want to code a `roc_auc_score` function to support multi-label output.

user: arjoly
issue_2381: state: open
title: univariate selection API issues
body: I think the univariate selection API is a bit unfriendly and doesn't follow the project philosophy (flat is better than nested).

My main concern is that we shouldn't require the user to import callables such as f_classif, f_regression and chi2. Just using a string would do the job. As far as I can see, f_regression is the only function with an option ("center"). It can be made an option of the feature selector.

I find the names f_classif and f_regression quite inexpressive. Are there better alternatives? 

Also I noticed that there is a GenericUnivariateSelect class but it's not documented in the user guide. 

user: mblondel
issue_2242: state: open
title: Common tests for feature  selection
body: ~~Maybe using `_LearntSelectorMixin`~~ That doesn't apply any more, but we still might want to do some basic testing.

user: amueller
labels: module:test-suite: everything related to our tests
issue_2121: state: open
title: Norm inconsistency between RFE and SelectFromModel
body: In each RFE iteration, the `step` features with the lowest importance are [discarded](https://github.com/djv/scikit-learn/blob/master/sklearn/feature_selection/rfe.py#L158). A similar thing [happens](https://github.com/djv/scikit-learn/blob/master/sklearn/feature_selection/from_model.py#L100) in `SelectFromModel` (but selection is by threshold rather than by number of features). There are some inconsistencies:
- ~~the mixin [admits](https://github.com/djv/scikit-learn/blob/master/sklearn/feature_selection/from_model.py#L44) either `est.coef_` or `est.feature_importances_` as the basis of its calculation; RFE only `coef_`~~ fixed in #4496
- the mixin uses `np.abs(est.coef_)`, while RFE uses `safe_sqr(est.coef_)`. The result is the same for selecting features by quantity, as long as `coef_` is 1d, but where 2d, the sum is taken over axis 0, and the ordering under L1 and L2 norms may differ.

~~Should RFE support `feature_importances_`?~~ For `coef_.ndim == 2`, should `SelectFromModel` use `sqrt(sqr(coef_).sum(axis=0))`?

(@glouppe, I think.)

user: jnothman
labels: Easy: Well-defined and straightforward way to resolve
issue_1978: state: open
title: DOC explain when to use which metric
body: @glouppe [wrote](https://github.com/scikit-learn/scikit-learn/pull/1945#issuecomment-17707603):

> The [model evaluation page](http://scikit-learn.org/dev/modules/model_evaluation.html) in the narrative documentation is getting quite big and I have the impression that it is becoming a copy of the reference documentation. It lists a bunch of metrics, explains what they are, but not really what they are for. I think it would be valuable to add some insights for the users: in which cases one should prefer a metric over the other?

@arjoly replied:

> Yes, this is something missing.
> More examples are needed to illustrate the metrics module and others should rewritten to give more insight to our user. Moreover something similar to the [clustering metrics documentation](http://scikit-learn.org/dev/modules/clustering.html#clustering-evaluation) would be a great addition to the documentation.

user: jnothman
issue_1651: state: open
title: Memory scalability issues in univariate feature selection
body: Gensim has a re-implementation of `selectkbest` to fix memory scalability issues. Someone should review them and it integrate the fix back into the original sklearn codebase.

https://github.com/piskvorky/gensim/blob/a73c84e21aecd3cc77ba2d752912f73b712bc60a/gensim/models/selectkbest.py

Maybe @agramfort  you can have a look?

user: ogrisel
issue_1626: state: open
title: API Proposal: Genearlized Cross-Validation and Early Stopping
body: This is a proposal to to resolve two API issues in sklearn:
- Generalized Cross Validation
- Early stopping
## Why should we care about that?

With **generalized cross validation** I mean finding the best setting of some parameter without refitting the entire model. This is currently implemented for RFE and some linear models via a EstimatorCV. These don't work well together with GridSearchCV as might be required in a Pipeline or when more than one parameter needs to be found.
Also, a similar functionality would be great for other models like GradientBoosting (for n_estimators) and all tree-based methods (for max_depth).

With **early stopping** I mean saving computations when more computation doesn't improve the result. We don't have that yet but it would be a great (maybe even necessary) feature for SGD based methods and bagging methods (random forests and extra trees). Note that early stopping needs the use of a validation set to evaluate the model.
## How can we solve that?

Let's start with the **generalized cross validation**.
We need it to work together with GridSearchCV.
This will definitely require changes in both GridSearchCV and the estimators.
My idea: 
1. give the estimator an iterable of values you want to try, i.e. `max_depth=range(1, 10)`. During `fit` the estimator will fit in a way that it can produce predictions for all of these values.
2. When `predict` is called, the estimator will return a dict with keys the parameter values and values the prediction values for these parameters. (we could also add a new `predict_all` function but I'm not sure about that).

GridSearchCV could then simply incooperate these values into the grid-search result.
For that GridSearchCV needs to be able to ask the estimator for which parameters it can do generalized CV and just pass  on the list of parameters it got there.

So now to **early stopping**. The reason I want to treat the two problems as one is that early stopping is basically a lazy form of generalized cross-validation.
So
1.  you would provide the estimator with an iterable, i.e. `n_iter=range(1, 100, 10)`.  
2. The estimator fits for all these values (as implemented as above). But for each setting, it also evaluates on a validation set, and if there is only a small improvement, training will stop.

I would provide the validation set that is used either as a parameter to  `__init__` or `fit` (not sure). So it would be enough to add two parameters to the estimator: `early_stopping_tolerance` and `early_stopping_set=None` (if it is None, no early stopping).

There are two choices that I made here:
1. provide a separate validation set, not generate one from the training set on the fly.
This is again so that this can be used inside GridSearchCV. And doesn't really add that much overhead if the user doesn't use GridSearchCV (why would they do that any way?)
It is also very explicit and gives the user a lot of control.
2. Provide the parameter settings as an iterable, not a maximum. The reason for that is that you probably don't want to evaluate the validation set every iteration, but maybe every k iterations. What's k? another parameter? I feel like an iterable is a good way to specify this. Also, it allows for a unified interface with the generalized cross-validation case.
## Restrictions

In pipelines, this will only work for the last estimator. So I'm not sure to do this with RFE for example.
## Do we really need this?

The changes I proposed above are quite big in some sense, but I think the two issues **need** to be resolved. If you have any better idea, feel free to explain it ;)

user: amueller
issue_1572: state: open
title: Common tests: check that results on sparse and dense matrices are identical
body: as proposed by zxtx :)

user: amueller
labels: module:test-suite: everything related to our tests
issue_1531: state: open
title: Implement faster ARDRegression
body: following http://books.nips.cc/papers/files/nips20/NIPS2007_0976.pdf

user: agramfort
issue_1453: state: open
title: MDS: fall back to SVD when the full similarity matrix is known
body: Right now, the MDS uses a slow majorization optimization method instead of falling back to the SVD when the full matrix is known. 

user: NelleV
issue_1362: state: open
title: Cross validation and tools for undersampling and oversampling
body: In learning with unbalanced classes, it is often good to stratify the data (not as in StratifiedKFold) by making the classes the same size, either by oversampling or undersampling.
I think it would be good to have both a utility function that does that once, and a cross-validation object that does that differently for all folds.

user: amueller
issue_1327: state: open
title: Precompute square norms in Passive-Aggressive
body: `PassiveAggressiveClassifier` is currently 5 times slower than `Perceptron` on the document classification example. This is because the current implementation recomputes the square norm (which is needed to compute the step size) for each example every time.

The solution would be to add a `void precompute_norms(squared=True)` and `double get_norm(int i)` to `SequentialDataset`. Doing so, I expect the performance to be inline with `Perceptron`.

CC @pprett 

user: mblondel
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_1261: state: open
title: Adaptive Learning Rate for SGD [enhancement]
body: I'm considering implementing a new adaptive learning rate algorithm proposed here:

http://arxiv.org/abs/1206.1106

Anyone have any feedback about where this should go?  I was thinking it would be added somewhere around https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/linear_model/sgd_fast.pyx#L383

Since SGD is also applicable to non-linear models (e.g. neural network models), are there any plans to move that outside of linear_model?

user: briancheung
issue_1183: state: open
title: Proposal to decouple classifier output classes from the y vector with a classes= parameter in __init__
body: From the discussion on the mailing list: http://sourceforge.net/mailarchive/message.php?msg_id=29883772

"I think we could have `classes=None` constructor parameter in
SGDClassifier an possibly many other classifiers. When provided we
would not use the traditional `self.classes_ = np.unique(y)` idiom
already implemented in some classifiers of the project (but not all).

+1 also for raising a ValueError exception when `classes != None` and
if the `y` provided at fit time has some values not in `classes`.
However we need to check with some benchmarks that this integrity
check is not too costly.

This constructor parameters could be overriden by a `fit_param` to
preserve backward compat, especially for classifier models with a
`partial_fit` method.

The expected behavior for a classifier that is passed a non-None
`classes` constructor param would be to never predict a class value.
In case of predict_proba method the missing fit-time class
probabilities should be 0.0.

This protocol (including expected exception types and error messages)
should be formalized as a series of common tests in
sklearn/tests/test_common.py and redundant book keeping code should be
factorized in the sklearn.base.py's ClassifierMixin class IMHO."

-Oliver Grisel

user: erg
labels: Needs Decision - API: Requires decision regarding API
issue_1179: state: open
title: GridSearchCV: allow for passing extra vectors to score/loss function?
body: There are situations when having vectors y_pred and y_true is not sufficient to compute the score/loss function. An easy example would be a mean (squared / absolute) error weighted by a pre-defined column of weights. In this case, this column vector has to be sliced by the cross-validator the same way X and y are sliced, and the sliced vector is to be passed to the score/loss function. While I understand that GridSearchCV can be (fairly painlessly) modified to account for this (say, an optional parameter to the score/loss func), I was wondering if anyone else ever faced the same problem.

user: aldanor
issue_1174: state: open
title: Implement Parallelized SGD as in NIPS 2010 paper
body: See http://cs.markusweimer.com/pub/2010/2010-NIPS.pdf

This will be "trivial" to implement efficiently once we have proper support for shared memory (both for the coef vector and the data) in joblib: joblib/joblib#44

user: ogrisel
issue_1091: state: open
title: Implement cluster stability measure
body: This is probably the only decent unsupervised clustering evaluation.
See here: http://arxiv.org/abs/1007.1075
Also this one: http://www.kyb.mpg.de/fileadmin/user_upload/files/publications/attachments/BenLuxPal06_%5B0%5D.pdf

user: amueller
issue_848: state: open
title: Add Kmeans parameter for pruning small clusters
body: In Kmeans, often some clusters have only very little data. This might happen for all random initializations.
For this case, I would like to have an option to set a minimum cluster size, after which a cluster is dropped and a new one is created.

user: amueller
labels: Moderate: Anything that requires some knowledge of conventions and best practices
issue_769: state: open
title: Iterative solvers for Ridge and warm_start
body: Iterative solvers for `Ridge` would allow to support the `warm_start` constructor option.

Relevant links:

http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.cg.html#scipy.sparse.linalg.cg

http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_cg.html#scipy.optimize.fmin_cg

user: mblondel
issue_735: state: open
title: Fixup naive_bayes
body: The `naive_bayes` needs some love!
1. <del>The current implementation of ``BaseDiscreteNB`` requires that the inputs are positive because otherwise the log computation in line 275 will fail. If ``BaseDiscreteNB`` is only applicable to non-negative inputs we should enforce that.</del>
2. <del>``GaussianNB`` what if the variance of a feature is zero? I did a quick fix and add a small constant epsilon (=1e-9). </del>
3. `GaussianNB` and `BaseDiscreteNB` should provide similar functionality. E.g. `GaussianNB` lacks a fit parameter `class_prior` and `class_weight`.
4. `GaussianNB` has a performance regression on `bench_covtype` - error rate used to be 0.23 now 0.46 - this needs to be investigated.
5. `GaussianNB` has poor test time performance - should be similar to `SGDClassifier` (use fortran-style for model parameters).

user: pprett
issue_638: state: open
title: Intercept sign in SVM needs love
body: I made an ugly hack in #634 to fix a regression I introduced earlier.
I would prefer a prettier solution.

user: amueller
issue_624: state: open
title: What would be the best way to add precomputed kernel support in RidgeCV ?
body: I'd like to be able to use precomputed kernels with RidgeCV. Would you be interested in this? If so, I'd like to hear your thoughts on how to do this well so that I can contribute a PR.

Related question: when `n_samples > n_features`, there is a more efficient way to get RidgeCV "LOOE" for free (with an SVD of `X`). Again would you be interested in this? How should I factor the code to contribute?

And thanks again for such a great package !

n

user: npinto
issue_555: state: open
title: Implement Adaptive Lasso / reweighted L1
body: Here is a proof of concept implementation by @agramfort 

   https://gist.github.com/1610922

Implementing this in the scikit would require a conversion to the scikit-learn API / naming conventions + example comparing with classic Lasso + narrative doc.

What motivated @agramfort's gist was:

  http://books.nips.cc/papers/files/nips24/NIPS2011_1135.pdf

See references therein for the original contributions.

user: ogrisel
issue_516: state: open
title: sparse-sparse and sparse-dense dot product with dense output
body: `safe_sparse_dot` has an option `dense_output` which allows to specify that the dot product between two sparse matrices or between a sparse matrix and a numpy array should be output to a numpy array. However, scipy currently always return a sparse matrix, therefore `safe_sparse_dot` converts it afterwards with `toarray()`. It would be better if we had Cython utilities that can directly output to a numpy array, thus avoiding needless memory allocation and copying. Those utilities are fairly low level and should probably be contributed to scipy too but I feel we should have them in scikit-learn, until the version of scipy that implements them becomes mainstream.

Used in:
- euclidean_distances
- linear_kernel
- ridge regression (n_features > n_samples case)
- NMF

Necessary functions for sparse-dense:
- dot(X_csr, Y_c, out)
- dot(X_csr, Y_fortran, out)

Necessary functions for sparse-sparse:
- dot(X_csr, Y_csc, out)

Other situations can be handled by using:
- dot(a, b) = dot(b.T, a.T).T

user: mblondel
issue_454: state: open
title: mclust for gmm
body: MCLUST (http://www.stat.washington.edu/mclust/) is the standard package for mixture modeling in the stats world. essentially, it is doing model based clustering as described in Fraley and Raftery, 2002 (www.stat.washington.edu/raftery/Research/PDF/fraley2002.pdf).  this means gaussian mixture modeling, essentially.  it has a number of choices for regularizing the covariance matrix.  it then uses BIC to choose.  the current implementation of gmm in sklearn has several of these options (full, diag, tied, spherical), but that is only about half the options in MCLUST.  and then while i see BIC is implemented for LassoLars, it seems to be tied to that.  

so, i wonder whether there are plans to:
1) update gmm.py to include other versions of covariance regularization
2) update model selection to include BIC generally (i couldn't find the src for the BIC implementation, so i can't tell how BIC is implemented now).

if no plans exist, i might fork.  PyMix has some model selection using BIC (http://pymix.org/docu/mixture-pysrc.html#modelSelection), so some of that code could be borrowed.....

thanks all for building such useful stuff!

user: jovo
issue_442: state: open
title: Add Gaussian kernel to mean shift clustering
body: # Motivation

Currently the mean shift clustering uses a flat kernel.

While this is quick and simple, there are some disadvantages.  There is no guarantee that the center will be the densest part a flat kernel---it might be that the center is sparse and that a single kernel covers the centers of two or more clusters at its periphery (If you mess around a bit with the test case by changing the bandwidth or seeding method, this disturbing behavior comes out---the center of the found clusters are in fact _between_ the true cluster centers).

I propose that we should also implement a Gaussian kernel because such a kernel would prefer to have dense regions near the center (and this is a common kernel in the literature).  The bandwidth parameter would act as the standard deviation for the gaussian kernel.  I propose that we use the same standard deviation in all dimensions (just as we used the same radius in all dimensions for the flat kernel).
# Efficiency considerations

The flat kernel is efficient because only points with a certain radius (specified by the bandwidth parameter) are relevant for the mean shift calculation.  These points can be efficiently looked up from the larger dataset using the BallTree.

In theory, each time we update a Gaussian kernel we should also consider _all_ points in the dataset, but in practice the method may work well if we ignore all points beyond some distance--let's call this the "cutoff distance".  Given that we expect over 99% of points to fall within three standard deviations of the center of a gaussian distribution, perhaps we could set this cutoff to 3X the bandwidth.

user: conradlee
issue_399: state: open
title: Bayesian priors in nearest neighbors classification/regression
body: Currently the classification and regression algorithms in the `neighbors` module use a flat prior.  They should be modified to compute a prior based on training data, and to optionally accept a user-defined prior.

user: jakevdp
issue_78: state: open
title: Use python logging to report on convergence progress it level info for long running tasks
body: This is a proposal to use python's logging module instead of using stdout and verbose flags in the models API.

Using the logging module would make it easier for the user to control the verbosity of the scikit using a single and well documented configuration interface and logging API.

http://docs.python.org/library/logging.html

user: ogrisel