Talip7/scikit-learn-issues-multilabel huggingface dataset:
sklearn implementation of the square-root LASSO

### Describe the workflow you want to enable

The unconstrained LASSO is currently implemented in `lasso_path`. However, the parameter for unconstrained LASSO is sensitive to the measurement noise. Instead, the square-root LASSO has objective norm(Ax-b) + alpha * norm(x, 1) and its parameter, alpha, is insensitive to the noise level. I would like an `sklearn` interface (with `fit`, `predict`, etc.) coupled with a `lasso_path`-like function, `srlasso_path`, which solves the square-root LASSO optimization program. 

### Describe your proposed solution

I propose an `sklearn`-friendly wrapper around the `cvxpy` or `cvxopt` interface. More ambitiously, I propose an implementation based on [1] that implements a fast (accelerated?) proximal gradient method for solving square-root LASSO.

[1] "On fast convergence of proximal algorithms for SQRT-LASSO optimization: Don’t worry about its nonsmooth loss function" by Li, Xinguo,  Jiang, Haoming, et al. (2020) PMLR. 

### Describe alternatives you've considered, if relevant

Currently, `statsmodels` offers a square-root LASSO solver implemented via the Python package `cvxopt` - this approach is slow. Using `cvxpy` with another interior point solver like MOSEK offers only 2x speed-up (it is snails compared to `lasso_path`). 

Separately, if the problem is appropriately regularized to avoid over-fitting, some work has shown that it's possible to use accelerated proximal gradient descent to achieve convergence. However, the paper offers an algorithm with several critical implementation details unprovided. See [1] above.

### Additional context

_No response_

COMMENT:
@asberk I'll close and suggest to follow @agramfort's proposal: Open a new issue for this feature in https://github.com/scikit-learn-contrib/skglm.

Labels: New Feature, module:linear_model, Needs Triage
Incremental versions of Existing Clustering Indices (metrics)

### Describe the workflow you want to enable

There currently exists 3 CVIs (cluster validity indices) in the main branch. These metrics are very useful for indicating performance of unsupervised clustering methods and add value to the module. However, these methods suffer from high computational complexity which can make them intractable for large datasets and online implementations. Silhouette, Davies-Bouldin, and Calinski-Harabasz have O(n^2), O(nlogn), and O(nlogn) complexity, respectively. 

Within the past few years, incremental versions of these algorithms have been introduced. These iCVIs replicate identical results of the batch (original) versions but in an online and more efficient way. The incremental versions of those existing CVIs have time complexities of O(n) for each. This has obvious benefit for large datasets and online implementation. 

While the incremental versions of these indices are themselves new and not highly cited, they are simply improvements to methods which already exist in this repository and would extend viability to a larger range of applications. 

### Describe your proposed solution

Implement incremental versions of the Silhouette, Davies-Bouldin, and Calinski-Harabasz methods within the metrics submodule. Each method would necessarily be a class with an incremental update method.

Given the go-ahead, I will submit a pull request with these methods.

### Describe alternatives you've considered, if relevant

Simply continue using the batch versions and deal with the time complexity

### Additional context

https://arxiv.org/pdf/1801.02937.pdf
https://ieeexplore.ieee.org/document/8970493

COMMENT:
Here is the current development repository/package for the batch and incremental CVI implementations: https://pypi.org/project/cvi/

Labels: New Feature
sklearn.compose.ColumnTransformer do not keep transformers` desired dtype of output

Hi,

We are using ColumnTransformer as our unified preprocessor to transform the data. We have the following transformers:

```python
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, FunctionTransformer



mean_imputer_pipline = Pipeline(steps=[('imputer', SimpleImputer(strategy='mean'))])
constan_one_imputer_pipline = Pipeline(steps=[('imputer', SimpleImputer(strategy='constant', fill_value=1.0))])
constan_zero_imputer_pipline = Pipeline(steps=[('imputer', SimpleImputer(strategy='constant', fill_value=0.0))])

categorical_transformer = Pipeline(steps=[
        ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
        ('onehot', OneHotEncoder(handle_unknown='ignore', dtype=np.int8))])

ordinal_categories = [determine_mapping(ordinal_feature) for ordinal_feature in ordinal_features]

ordinal_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
                                          ('ordinal', OrdinalEncoder(categories=ordinal_categories, dtype=np.int8))])
```

From them we create our ColumnTransformer object:
```python
from sklearn.compose import ColumnTransformer



preprocessor = ColumnTransformer(
        transformers=[
            ('num_mean', mean_imputer_pipline, mean_imputer_features),
            ('num_constant_one', constan_one_imputer_pipline, constant_one_imputer_features),
            ('num_constant_zero', constan_zero_imputer_pipline, constant_zero_imputer_features),
            ('cat', categorical_transformer, categorical_features),
            ('ordinal', ordinal_transformer, ordinal_features)],
        remainder="drop")
```

When we use the ```transform``` method the desired dtypes of output for  ```categorical_transformer``` and ```ordinal_transformer``` that should be ```np.int8``` isn't kept. What we are receiving is numpy's ndarray that its dtype is ```np.float64```.

We debuged the code, and found out that in order to join the outputs from the different transformers the ColumnTransformer is using the method ```np.hstack``` which creates a unified 1 ndarray with one dtype.

Is there a solution?
If not:
1. Please add a warning in the documentation of the ColumnTransformer that the concatenate results won't use the desired dtype of output of the transformers. 
2. Allow us an option to receive the result without concatenation, as a list of ndarrays.

COMMENT:
Thanks @thomasjpfan for the quick response.

So right now we have no other way but to run the transformers one by one?

When are expecting to see the solution of slep018 in scikit-learn?

Labels: Documentation, module:compose
sklearn.compose.ColumnTransformer do not keep transformers` desired dtype of output

Hi,

We are using ColumnTransformer as our unified preprocessor to transform the data. We have the following transformers:

```python
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, FunctionTransformer



mean_imputer_pipline = Pipeline(steps=[('imputer', SimpleImputer(strategy='mean'))])
constan_one_imputer_pipline = Pipeline(steps=[('imputer', SimpleImputer(strategy='constant', fill_value=1.0))])
constan_zero_imputer_pipline = Pipeline(steps=[('imputer', SimpleImputer(strategy='constant', fill_value=0.0))])

categorical_transformer = Pipeline(steps=[
        ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
        ('onehot', OneHotEncoder(handle_unknown='ignore', dtype=np.int8))])

ordinal_categories = [determine_mapping(ordinal_feature) for ordinal_feature in ordinal_features]

ordinal_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
                                          ('ordinal', OrdinalEncoder(categories=ordinal_categories, dtype=np.int8))])
```

From them we create our ColumnTransformer object:
```python
from sklearn.compose import ColumnTransformer



preprocessor = ColumnTransformer(
        transformers=[
            ('num_mean', mean_imputer_pipline, mean_imputer_features),
            ('num_constant_one', constan_one_imputer_pipline, constant_one_imputer_features),
            ('num_constant_zero', constan_zero_imputer_pipline, constant_zero_imputer_features),
            ('cat', categorical_transformer, categorical_features),
            ('ordinal', ordinal_transformer, ordinal_features)],
        remainder="drop")
```

When we use the ```transform``` method the desired dtypes of output for  ```categorical_transformer``` and ```ordinal_transformer``` that should be ```np.int8``` isn't kept. What we are receiving is numpy's ndarray that its dtype is ```np.float64```.

We debuged the code, and found out that in order to join the outputs from the different transformers the ColumnTransformer is using the method ```np.hstack``` which creates a unified 1 ndarray with one dtype.

Is there a solution?
If not:
1. Please add a warning in the documentation of the ColumnTransformer that the concatenate results won't use the desired dtype of output of the transformers. 
2. Allow us an option to receive the result without concatenation, as a list of ndarrays.

COMMENT:
That's a very good explanation @thomasjpfan  even I too faced this issue . Hopefully in SLEP018 we could find an alternate solution.

Labels: Documentation, module:compose
sklearn.compose.ColumnTransformer do not keep transformers` desired dtype of output

Hi,

We are using ColumnTransformer as our unified preprocessor to transform the data. We have the following transformers:

```python
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, FunctionTransformer



mean_imputer_pipline = Pipeline(steps=[('imputer', SimpleImputer(strategy='mean'))])
constan_one_imputer_pipline = Pipeline(steps=[('imputer', SimpleImputer(strategy='constant', fill_value=1.0))])
constan_zero_imputer_pipline = Pipeline(steps=[('imputer', SimpleImputer(strategy='constant', fill_value=0.0))])

categorical_transformer = Pipeline(steps=[
        ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
        ('onehot', OneHotEncoder(handle_unknown='ignore', dtype=np.int8))])

ordinal_categories = [determine_mapping(ordinal_feature) for ordinal_feature in ordinal_features]

ordinal_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
                                          ('ordinal', OrdinalEncoder(categories=ordinal_categories, dtype=np.int8))])
```

From them we create our ColumnTransformer object:
```python
from sklearn.compose import ColumnTransformer



preprocessor = ColumnTransformer(
        transformers=[
            ('num_mean', mean_imputer_pipline, mean_imputer_features),
            ('num_constant_one', constan_one_imputer_pipline, constant_one_imputer_features),
            ('num_constant_zero', constan_zero_imputer_pipline, constant_zero_imputer_features),
            ('cat', categorical_transformer, categorical_features),
            ('ordinal', ordinal_transformer, ordinal_features)],
        remainder="drop")
```

When we use the ```transform``` method the desired dtypes of output for  ```categorical_transformer``` and ```ordinal_transformer``` that should be ```np.int8``` isn't kept. What we are receiving is numpy's ndarray that its dtype is ```np.float64```.

We debuged the code, and found out that in order to join the outputs from the different transformers the ColumnTransformer is using the method ```np.hstack``` which creates a unified 1 ndarray with one dtype.

Is there a solution?
If not:
1. Please add a warning in the documentation of the ColumnTransformer that the concatenate results won't use the desired dtype of output of the transformers. 
2. Allow us an option to receive the result without concatenation, as a list of ndarrays.

COMMENT:
Hey @thomasjpfan
Any news?

Labels: Documentation, module:compose
sklearn.compose.ColumnTransformer do not keep transformers` desired dtype of output

Hi,

We are using ColumnTransformer as our unified preprocessor to transform the data. We have the following transformers:

```python
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, FunctionTransformer



mean_imputer_pipline = Pipeline(steps=[('imputer', SimpleImputer(strategy='mean'))])
constan_one_imputer_pipline = Pipeline(steps=[('imputer', SimpleImputer(strategy='constant', fill_value=1.0))])
constan_zero_imputer_pipline = Pipeline(steps=[('imputer', SimpleImputer(strategy='constant', fill_value=0.0))])

categorical_transformer = Pipeline(steps=[
        ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
        ('onehot', OneHotEncoder(handle_unknown='ignore', dtype=np.int8))])

ordinal_categories = [determine_mapping(ordinal_feature) for ordinal_feature in ordinal_features]

ordinal_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
                                          ('ordinal', OrdinalEncoder(categories=ordinal_categories, dtype=np.int8))])
```

From them we create our ColumnTransformer object:
```python
from sklearn.compose import ColumnTransformer



preprocessor = ColumnTransformer(
        transformers=[
            ('num_mean', mean_imputer_pipline, mean_imputer_features),
            ('num_constant_one', constan_one_imputer_pipline, constant_one_imputer_features),
            ('num_constant_zero', constan_zero_imputer_pipline, constant_zero_imputer_features),
            ('cat', categorical_transformer, categorical_features),
            ('ordinal', ordinal_transformer, ordinal_features)],
        remainder="drop")
```

When we use the ```transform``` method the desired dtypes of output for  ```categorical_transformer``` and ```ordinal_transformer``` that should be ```np.int8``` isn't kept. What we are receiving is numpy's ndarray that its dtype is ```np.float64```.

We debuged the code, and found out that in order to join the outputs from the different transformers the ColumnTransformer is using the method ```np.hstack``` which creates a unified 1 ndarray with one dtype.

Is there a solution?
If not:
1. Please add a warning in the documentation of the ColumnTransformer that the concatenate results won't use the desired dtype of output of the transformers. 
2. Allow us an option to receive the result without concatenation, as a list of ndarrays.

COMMENT:
> An upcoming solution to this will be https://github.com/scikit-learn/enhancement_proposals/pull/72 which will enable pandas output for transformers, which includes ColumnTransformer. This will preserve the dtypes by returning a pandas DataFrame.

SLEP018 Didn't resolve this issue, even when using `set_output(transform='pandas")` dtype is lost

Labels: Documentation, module:compose
AgglomerativeClustering's distance_threshold is misleading

### Describe the issue linked to the documentation

According to the [documentation regarding `AgglomerativeClustering`](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html#sklearn.cluster.AgglomerativeClustering), the `distance_threshold` is "the linkage distance threshold above which, clusters will not be merged".

However, in the [code](https://github.com/scikit-learn/scikit-learn/blob/36958fb24/sklearn/cluster/_agglomerative.py#L1014), values at the exact same distance as `distance_threshold` are also not merged.

### Suggest a potential alternative/fix

I would suggest making it clear that the `distance_threshold` is the linkage distance threshold where clusters will not be merged if a distance is equal to or above the given value. Another solution could be changing the code so that it uses `>` instead of `>=`, but that might not be as welcomed.

COMMENT:
I'm in favor of improving the wording to, perhaps "...at or above which..."

@joaocmd Would you be interested in opening a PR for this?

Labels: Documentation, help wanted, module:cluster
AgglomerativeClustering's distance_threshold is misleading

### Describe the issue linked to the documentation

According to the [documentation regarding `AgglomerativeClustering`](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html#sklearn.cluster.AgglomerativeClustering), the `distance_threshold` is "the linkage distance threshold above which, clusters will not be merged".

However, in the [code](https://github.com/scikit-learn/scikit-learn/blob/36958fb24/sklearn/cluster/_agglomerative.py#L1014), values at the exact same distance as `distance_threshold` are also not merged.

### Suggest a potential alternative/fix

I would suggest making it clear that the `distance_threshold` is the linkage distance threshold where clusters will not be merged if a distance is equal to or above the given value. Another solution could be changing the code so that it uses `>` instead of `>=`, but that might not be as welcomed.

COMMENT:
> @joaocmd Would you be interested in opening a PR for this?

Created a PR now. Have noticed now that @rusdes had already opened a similar one. However, it seems like they missed one reference to that issue. Will reference it on my PR nonetheless.

Labels: Documentation, help wanted, module:cluster
SGDClassifier initial lr undocumented

### Describe the issue linked to the documentation

The [documentation for SGDClassifier](https://scikit-learn.org/stable/modules/sgd.html#sgd) says that with `learning_rate='optimal'` the initial learning rate `t0` is chosen with a heuristic proposed by Léon Bottou that "can be found in `_init_t` in BaseSGD".

However currently this method [does not exist in BaseSGD](https://github.com/scikit-learn/scikit-learn/search?q=_init_t).

### Suggest a potential alternative/fix

The calculation of t0 should be documented either by adding the formula, pointing to the correct method in the code, and/or specifying exactly what paper of Bottou is being referred to.

COMMENT:
Hi @cmarmo / @Micky774  can i work on this issue?

Labels: Documentation, module:linear_model
SGDClassifier initial lr undocumented

### Describe the issue linked to the documentation

The [documentation for SGDClassifier](https://scikit-learn.org/stable/modules/sgd.html#sgd) says that with `learning_rate='optimal'` the initial learning rate `t0` is chosen with a heuristic proposed by Léon Bottou that "can be found in `_init_t` in BaseSGD".

However currently this method [does not exist in BaseSGD](https://github.com/scikit-learn/scikit-learn/search?q=_init_t).

### Suggest a potential alternative/fix

The calculation of t0 should be documented either by adding the formula, pointing to the correct method in the code, and/or specifying exactly what paper of Bottou is being referred to.

COMMENT:
> Hi @cmarmo / @Micky774 can i work on this issue?

Sure! I think improving the existing documentation to either explain or at least present the heuristic is probably the best way to proceed.

Labels: Documentation, module:linear_model
SGDClassifier initial lr undocumented

### Describe the issue linked to the documentation

The [documentation for SGDClassifier](https://scikit-learn.org/stable/modules/sgd.html#sgd) says that with `learning_rate='optimal'` the initial learning rate `t0` is chosen with a heuristic proposed by Léon Bottou that "can be found in `_init_t` in BaseSGD".

However currently this method [does not exist in BaseSGD](https://github.com/scikit-learn/scikit-learn/search?q=_init_t).

### Suggest a potential alternative/fix

The calculation of t0 should be documented either by adding the formula, pointing to the correct method in the code, and/or specifying exactly what paper of Bottou is being referred to.

COMMENT:
Thanks!

The default value for `t_0` is not explicitly defined but is highlighted within https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/linear_model/_sgd_fast.pyx#L526
```
    if learning_rate == OPTIMAL:
        typw = np.sqrt(1.0 / np.sqrt(alpha))
        # computing eta0, the initial learning rate
        initial_eta0 = typw / max(1.0, loss.dloss(-typw, 1.0))
        # initialize t such that eta at first sample equals eta0
        optimal_init = 1.0 / (initial_eta0 * alpha)
```
As the documentation states for `alpha` - yes there is a grammatical error within the docs:
```
alpha` : ..._Also used to compute the learning rate when set to learning_rate is set to ‘optimal’. Values must be in the range [0.0, inf).  default=0.0001
```

Later on the in code:

https://github.com/scikit-learn/scikit-learn/blob/14130f44eb6cba8a2fb2eff8383be8909783cad0/sklearn/linear_model/_sgd_fast.pyx#L553

`t=1` by default so `t_0` would be 0. Not extremely familiar with the Plain SGD formula but with some rearranging of the formula within the code it looks like plain SGD formula within: https://leon.bottou.org/projects/sgd 
This is reference [7] on the doc page. 

Am I missing something?

Labels: Documentation, module:linear_model
SGDClassifier initial lr undocumented

### Describe the issue linked to the documentation

The [documentation for SGDClassifier](https://scikit-learn.org/stable/modules/sgd.html#sgd) says that with `learning_rate='optimal'` the initial learning rate `t0` is chosen with a heuristic proposed by Léon Bottou that "can be found in `_init_t` in BaseSGD".

However currently this method [does not exist in BaseSGD](https://github.com/scikit-learn/scikit-learn/search?q=_init_t).

### Suggest a potential alternative/fix

The calculation of t0 should be documented either by adding the formula, pointing to the correct method in the code, and/or specifying exactly what paper of Bottou is being referred to.

COMMENT:
I think the intention is to essentially include and explain the first block of code into to the documentation. That is, 
```python
    if learning_rate == OPTIMAL:
        typw = np.sqrt(1.0 / np.sqrt(alpha))
        # computing eta0, the initial learning rate
        initial_eta0 = typw / max(1.0, loss.dloss(-typw, 1.0))
        # initialize t such that eta at first sample equals eta0
        optimal_init = 1.0 / (initial_eta0 * alpha)
``` 
should be documented in the User Guide, since `t_0 := optimal_init`.

Labels: Documentation, module:linear_model
SGDClassifier initial lr undocumented

### Describe the issue linked to the documentation

The [documentation for SGDClassifier](https://scikit-learn.org/stable/modules/sgd.html#sgd) says that with `learning_rate='optimal'` the initial learning rate `t0` is chosen with a heuristic proposed by Léon Bottou that "can be found in `_init_t` in BaseSGD".

However currently this method [does not exist in BaseSGD](https://github.com/scikit-learn/scikit-learn/search?q=_init_t).

### Suggest a potential alternative/fix

The calculation of t0 should be documented either by adding the formula, pointing to the correct method in the code, and/or specifying exactly what paper of Bottou is being referred to.

COMMENT:
Thanks for clarifying @Micky774 

Doing a bit of a dive in the existing references I was able to confirm the formula that is being used. However, the hyperparameter `alpha` is not defined in the paper nor in any of the formulas. `alpha` would be `0.0001` by default and there isn't anything in the SGD module that makes me think this is calculated a different way, unless you are aware? Nonetheless, it clearly is a regularising term and allows scheduling of learning rates that are asymptotically decreasing.

From what I can gather, the initial optimal learning rate would be given by:

```math
\eta_{(0)} = \frac {\alpha^{-0.25}}{max(1.0, \frac{\partial L(-\alpha^{-0.25})}{\partial w})}
```

Where `alpha` and the selected loss function are hyperparameters chosen by the user via arguments `alpha` and `loss`, respectively.

Is this the level at which the User Guide should be, or should I try to find the reference where the above formula was taken from?

Labels: Documentation, module:linear_model