Talip7/scikit-learn-issues-multilabel huggingface dataset:
SGDClassifier initial lr undocumented

### Describe the issue linked to the documentation

The [documentation for SGDClassifier](https://scikit-learn.org/stable/modules/sgd.html#sgd) says that with `learning_rate='optimal'` the initial learning rate `t0` is chosen with a heuristic proposed by Léon Bottou that "can be found in `_init_t` in BaseSGD".

However currently this method [does not exist in BaseSGD](https://github.com/scikit-learn/scikit-learn/search?q=_init_t).

### Suggest a potential alternative/fix

The calculation of t0 should be documented either by adding the formula, pointing to the correct method in the code, and/or specifying exactly what paper of Bottou is being referred to.

COMMENT:
> ```python
>     if learning_rate == OPTIMAL:
>         typw = np.sqrt(1.0 / np.sqrt(alpha))
>         # computing eta0, the initial learning rate
>         initial_eta0 = typw / max(1.0, loss.dloss(-typw, 1.0))
>         # initialize t such that eta at first sample equals eta0
>         optimal_init = 1.0 / (initial_eta0 * alpha)
> ```

Could anyone explain why it use `alpha` -- the regularization parameter to set `t_0` like this? Thx.

Labels: Documentation, module:linear_model
SGDClassifier initial lr undocumented

### Describe the issue linked to the documentation

The [documentation for SGDClassifier](https://scikit-learn.org/stable/modules/sgd.html#sgd) says that with `learning_rate='optimal'` the initial learning rate `t0` is chosen with a heuristic proposed by Léon Bottou that "can be found in `_init_t` in BaseSGD".

However currently this method [does not exist in BaseSGD](https://github.com/scikit-learn/scikit-learn/search?q=_init_t).

### Suggest a potential alternative/fix

The calculation of t0 should be documented either by adding the formula, pointing to the correct method in the code, and/or specifying exactly what paper of Bottou is being referred to.

COMMENT:
Here's an overview of one commonly used approach by the paper:

Set a default learning rate range: Define a reasonable range for the initial learning rate, such as [0.01, 0.1]. 

Conduct a coarse grid search: Start by selecting a few learning rate values within the defined range, for example, [0.01, 0.03, 0.1]. Train the model using each learning rate for a small number of iterations or epochs (e.g., 10-20) and record the corresponding performance metrics (e.g., loss, accuracy). -> This could be alpha

Analyze the performance: Examine the model's performance at each learning rate and observe how the performance changes. Look for learning rates that result in fast convergence without causing instability or poor performance.

Refine the search: Based on the results of the coarse grid search, narrow down the range of learning rates to a smaller interval that shows promising performance. For example, if [0.01, 0.03, 0.1] yielded good results, refine the range to [0.01, 0.02, 0.03].

Perform a finer grid search: Repeat the training process with the refined learning rate range, using a higher number of iterations or epochs. This helps to obtain a more accurate assessment of the model's performance at each learning rate.

Choose the optimal learning rate: Analyze the results of the finer grid search and select the learning rate that leads to the best overall performance. This could be the learning rate that achieves the fastest convergence or the highest accuracy, depending on the specific goals of your task.

It's important to note that the process of finding the initial learning rate can be iterative and may require experimentation and adjustment based on the characteristics of the dataset and model. The goal is to find a learning rate that enables stable and efficient convergence during training, leading to good generalization and performance on unseen data.

We can add some of the following points in the documentation, Thoughts @adrinjalali

Labels: Documentation, module:linear_model
SGDClassifier initial lr undocumented

### Describe the issue linked to the documentation

The [documentation for SGDClassifier](https://scikit-learn.org/stable/modules/sgd.html#sgd) says that with `learning_rate='optimal'` the initial learning rate `t0` is chosen with a heuristic proposed by Léon Bottou that "can be found in `_init_t` in BaseSGD".

However currently this method [does not exist in BaseSGD](https://github.com/scikit-learn/scikit-learn/search?q=_init_t).

### Suggest a potential alternative/fix

The calculation of t0 should be documented either by adding the formula, pointing to the correct method in the code, and/or specifying exactly what paper of Bottou is being referred to.

COMMENT:
Thanks to @ogrisel 's help, it seems this is coming from [this paper](https://leon.bottou.org/publications/pdf/tricks-2012.pdf), and particularly section 5.2 this section:

![image](https://github.com/scikit-learn/scikit-learn/assets/1663864/c6c313ff-38bc-4021-a392-5c099115b771)

Labels: Documentation, module:linear_model
⚠️ CI failed on Wheel builder ⚠️

**CI is still failing on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/2837268478)** (Aug 11, 2022)

COMMENT:
`test_estimators[SpectralEmbedding()-check_pipeline_consistency]` fails on Windows, maybe related to https://github.com/scikit-learn/scikit-learn/issues/24105 although this time rather that columns being flipped the column 0 seems seems slightly off while the column 1 seems to match:

```
actual = array([[ 0.11604132,  0.09269019],
       [-0.0414285 ,  0.16213956],
       [-0.0414285 ,  0.16213956],
       [ 0.32... 0.16213956],
       [ 0.11604132,  0.09269019],
       [ 0.11604132,  0.09269019],
       [-0.0414285 ,  0.16213956]])
desired = array([[-0.14003431,  0.09269019],
       [ 0.07299161,  0.16213956],
       [ 0.07299161,  0.16213956],
       [ 0.24... 0.16213956],
       [-0.14003431,  0.09269019],
       [-0.14003431,  0.09269019],
       [ 0.07299161,  0.16213956]])
```

I'll merge https://github.com/scikit-learn/scikit-learn/pull/24125 to see whether it fixes this error as well.

Labels: Needs Triage
⚠️ CI failed on Wheel builder ⚠️

**CI is still failing on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/2837268478)** (Aug 11, 2022)

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://github.com/scikit-learn/scikit-learn/actions/runs/2844366471) on Aug 12, 2022

Labels: Needs Triage
LogisticRegression doesn't have an offset option

### Describe the workflow you want to enable

Like in R, can we add an option of offset in sklearn so that we can fix the value of some coefficients for logistic regression?

For those who don't know offset, offset is defined as a fixed feature with coefficient being 1. Therefore, if $y=g(x\beta + z\alpha)$ and we want to choose some pre-determined value $\alpha_0$, we can just pass $z\alpha_0$ to offset. 

### Describe your proposed solution

I feel this might be straightforward but, as someone who doesn't know the original code/module that well, it would be great if someone can give any advice on if this is feasible. Also any suggestions on its functionality would be helpful.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hey there! If no one is working on it, can I take it up?

Labels: New Feature, module:linear_model, Needs Decision - Include Feature
LogisticRegression doesn't have an offset option

### Describe the workflow you want to enable

Like in R, can we add an option of offset in sklearn so that we can fix the value of some coefficients for logistic regression?

For those who don't know offset, offset is defined as a fixed feature with coefficient being 1. Therefore, if $y=g(x\beta + z\alpha)$ and we want to choose some pre-determined value $\alpha_0$, we can just pass $z\alpha_0$ to offset. 

### Describe your proposed solution

I feel this might be straightforward but, as someone who doesn't know the original code/module that well, it would be great if someone can give any advice on if this is feasible. Also any suggestions on its functionality would be helpful.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> Hey there! If no one is working on it, can I take it up?

Yes, please! I don't know well how it should work. But thanks for volunteering!

Labels: New Feature, module:linear_model, Needs Decision - Include Feature
LogisticRegression doesn't have an offset option

### Describe the workflow you want to enable

Like in R, can we add an option of offset in sklearn so that we can fix the value of some coefficients for logistic regression?

For those who don't know offset, offset is defined as a fixed feature with coefficient being 1. Therefore, if $y=g(x\beta + z\alpha)$ and we want to choose some pre-determined value $\alpha_0$, we can just pass $z\alpha_0$ to offset. 

### Describe your proposed solution

I feel this might be straightforward but, as someone who doesn't know the original code/module that well, it would be great if someone can give any advice on if this is feasible. Also any suggestions on its functionality would be helpful.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I know that this one was opened earlier, but I close as duplicate of #24614. So we can have the discussion in a single place instead of scattered.

Labels: New Feature, module:linear_model, Needs Decision - Include Feature
joblib and threadpoolctl dependencies are missing in setup.py and pyproject.toml

### Describe the bug

README.rst lists them as dependencies, but they aren't in setup.py and pyproject.toml

### Steps/Code to Reproduce

n/a

### Expected Results

n/a

### Actual Results

n/a

### Versions

```shell
n/a
```

COMMENT:
The dependencies are in `setup.py` through:

https://github.com/scikit-learn/scikit-learn/blob/f0cbdc15f5031f0595ffcc8f822e5318a2335349/setup.py#L276

which dynamically imports https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/_min_dependencies.py to get dependencies for install. Currently, `_min_dependencies` is the "single source of truth" for all our dependencies.

Strictly speaking, `pyproject.toml` only requires build dependencies and not runtime dependencies. During installation, the runtime dependencies will be installed through `setup.py`. 

For my understanding, why do you require the runtime dependencies in `pyproject.toml`?

Labels: Enhancement, Build / CI
joblib and threadpoolctl dependencies are missing in setup.py and pyproject.toml

### Describe the bug

README.rst lists them as dependencies, but they aren't in setup.py and pyproject.toml

### Steps/Code to Reproduce

n/a

### Expected Results

n/a

### Actual Results

n/a

### Versions

```shell
n/a
```

COMMENT:
I am not familiar with dynamic imports of install_requires, but that's fine if this works.

> For my understanding, why do you require the runtime dependencies in pyproject.toml?

Because ports should know what does the package depend on. For example, FreeBSD ports have RUN_DEPENDS line that has to be populated. If not from pyproject.toml, how can it be populated?

Labels: Enhancement, Build / CI
joblib and threadpoolctl dependencies are missing in setup.py and pyproject.toml

### Describe the bug

README.rst lists them as dependencies, but they aren't in setup.py and pyproject.toml

### Steps/Code to Reproduce

n/a

### Expected Results

n/a

### Actual Results

n/a

### Versions

```shell
n/a
```

COMMENT:
Before `pyproject.toml` existed, I think most package managers list out the build and runtime dependencies themselves by looking at the `README`. For example, [ubuntu's APT](https://packages.ubuntu.com/jammy/python3-sklearn) lists out scikit-learn's dependencies themselves and [Arch linux](https://archlinux.pkgs.org/rolling/archlinux-community-aarch64/python-scikit-learn-1.1.1-1-aarch64.pkg.tar.xz.html) does the same thing.

For some package managers, `pyproject.toml` may not be sufficient. For example, scikit-learn also requires a compiler, which can not be specified in `pyproject.toml`. [conda-forge](https://github.com/conda-forge/scikit-learn-feedstock/blob/2596bc52463b79f7228151c8c247c497be7e8ec3/recipe/meta.yaml#L31-L33) specifies the C and C++ compilers in it's build configuration. For conda-forge, we also need to manually update the [minimum versions](https://github.com/conda-forge/scikit-learn-feedstock/blob/2596bc52463b79f7228151c8c247c497be7e8ec3/recipe/meta.yaml#L51-L52) when scikit-learn bumps the minimum supported versions for our dependencies.

Also, the names in `pyproject.toml` references to package names on PyPI, which may not be consistent with the names on a package manager. For example, ubuntu's apt uses [python3-sklearn](https://packages.ubuntu.com/jammy/python3-sklearn) as the name for scikit-learn, while the name on PyPi is [scikit-learn](https://pypi.org/project/scikit-learn/). Arch Linux uses [python-scikit-learn](https://archlinux.pkgs.org/rolling/archlinux-community-aarch64/python-scikit-learn-1.1.1-1-aarch64.pkg.tar.xz.html). If those package mangers wanted programmatically use `pyproject.toml`, they would need to manage a mapping between the PyPi name and their packager name. They would also need to make sure that the dependency exist on their package manager.

For FreeBSD port, if a developer already needs to manually update the dependencies in their build system, then I think having the dependencies listed in scikit-learn's README is sufficient. The README is focused on displaying information to a human reader.

Labels: Enhancement, Build / CI
joblib and threadpoolctl dependencies are missing in setup.py and pyproject.toml

### Describe the bug

README.rst lists them as dependencies, but they aren't in setup.py and pyproject.toml

### Steps/Code to Reproduce

n/a

### Expected Results

n/a

### Actual Results

n/a

### Versions

```shell
n/a
```

COMMENT:
> Before pyproject.toml existed, I think most package managers list out the build and runtime dependencies themselves by looking at the README.

No. setup.py had/has tags install_require, test_require, etc.

Labels: Enhancement, Build / CI
joblib and threadpoolctl dependencies are missing in setup.py and pyproject.toml

### Describe the bug

README.rst lists them as dependencies, but they aren't in setup.py and pyproject.toml

### Steps/Code to Reproduce

n/a

### Expected Results

n/a

### Actual Results

n/a

### Versions

```shell
n/a
```

COMMENT:
> For FreeBSD port, if a developer already needs to manually update the dependencies in their build system, then I think having the dependencies listed in scikit-learn's README is sufficient.

This is a very uncommon behavior in scikit-learn. Most others list them in setup.py and pyproject.toml

Labels: Enhancement, Build / CI
joblib and threadpoolctl dependencies are missing in setup.py and pyproject.toml

### Describe the bug

README.rst lists them as dependencies, but they aren't in setup.py and pyproject.toml

### Steps/Code to Reproduce

n/a

### Expected Results

n/a

### Actual Results

n/a

### Versions

```shell
n/a
```

COMMENT:
> No. setup.py had/has tags install_require, test_require, etc.

scikit-learn already populates `install_require` in `setup.py` dynamically. When we build a wheel for PyPI, this metadata is injected into the wheel.

Labels: Enhancement, Build / CI
joblib and threadpoolctl dependencies are missing in setup.py and pyproject.toml

### Describe the bug

README.rst lists them as dependencies, but they aren't in setup.py and pyproject.toml

### Steps/Code to Reproduce

n/a

### Expected Results

n/a

### Actual Results

n/a

### Versions

```shell
n/a
```

COMMENT:
> This is a very uncommon behavior in scikit-learn. Most others list them in setup.py and pyproject.toml

I do not think there has been a common standard to specify this information. In other libraries, I have seen dependencies in `setup.cfg`, `setup.py`, `pyproject.toml`, and `requirements.txt` (which is dynamically loaded into `setup.py`).

That being said, I am in favor of moving dependencies and other information into `pyproject.toml`. `pyproject.toml` seems like what most of the Python ecosystem is moving toward.

Labels: Enhancement, Build / CI
DOC LeaveOneGroupOut description confusing

### Describe the issue linked to the documentation

The description for [`LeaveOneGroupOut`](https://scikit-learn.org/stable/modules/cross_validation.html#leave-one-group-out) in the user guide:

> :class:`LeaveOneGroupOut` is a cross-validation scheme which holds out
the samples according to a third-party provided array of integer groups. This
group information can be used to encode arbitrary domain specific pre-defined
cross-validation folds.

and in the docstring:

https://github.com/scikit-learn/scikit-learn/blob/87d182de2454c3690c944fa5da61cadb01eeef4f/sklearn/model_selection/_split.py#L1112-L1114

may be confusing to new comers when it uses the phrase "third-party provided" - who is this third party? Who are the first and second parties in this instance?

### Suggest a potential alternative/fix

The docs for the other grouped cross validation splitters don't use this term 'third-party' - we could use instead 'domain specific grouping of data samples' etc?

COMMENT:
I agree the term "third party" is confusing here. I am in favor with your suggestion of using "domain specific groups"

Labels: Documentation, module:model_selection
Add Hannan-Quinn IC to LassoLarsIC

### Describe the workflow you want to enable

Hannan–Quinn information criterion was originally described to be an alternative model selection criterion for autoregressive models, but it could be used in other cases too. Unlike AIC it also penalizes model's number of parameters, but unlike BIC it imposes a smaller penalty on complex models in large samples, which is beneficial when working with large amounts of complex data.

Some links: [wiki](https://en.wikipedia.org/wiki/Hannan%E2%80%93Quinn_information_criterion), [original paper](https://www.jstor.org/stable/2985032), [comparison of different IC for autoregressive models](http://www3.stat.sinica.edu.tw/statistica/oldpdf/A3n214.pdf), [model selection techniques overview](https://arxiv.org/pdf/1810.09583.pdf)

### Describe your proposed solution

Adding two lines of code to existing implementation:
            
`elif self.criterion == "hqic":`
    `    criterion_factor = 2*log(log(n_samples))`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

It shouldn't be too costly to maintain and will provide valid alternatives to AIC and BIC.

COMMENT:
Hi! I would like to work on this. Can I proceed?

Labels: New Feature, module:linear_model, Needs Decision - Include Feature
Add Hannan-Quinn IC to LassoLarsIC

### Describe the workflow you want to enable

Hannan–Quinn information criterion was originally described to be an alternative model selection criterion for autoregressive models, but it could be used in other cases too. Unlike AIC it also penalizes model's number of parameters, but unlike BIC it imposes a smaller penalty on complex models in large samples, which is beneficial when working with large amounts of complex data.

Some links: [wiki](https://en.wikipedia.org/wiki/Hannan%E2%80%93Quinn_information_criterion), [original paper](https://www.jstor.org/stable/2985032), [comparison of different IC for autoregressive models](http://www3.stat.sinica.edu.tw/statistica/oldpdf/A3n214.pdf), [model selection techniques overview](https://arxiv.org/pdf/1810.09583.pdf)

### Describe your proposed solution

Adding two lines of code to existing implementation:
            
`elif self.criterion == "hqic":`
    `    criterion_factor = 2*log(log(n_samples))`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

It shouldn't be too costly to maintain and will provide valid alternatives to AIC and BIC.

COMMENT:
This suggestion was not checked, discussed and green-lighted by the maintainers yet, so there is no need for PR right now.

Labels: New Feature, module:linear_model, Needs Decision - Include Feature
Add Hannan-Quinn IC to LassoLarsIC

### Describe the workflow you want to enable

Hannan–Quinn information criterion was originally described to be an alternative model selection criterion for autoregressive models, but it could be used in other cases too. Unlike AIC it also penalizes model's number of parameters, but unlike BIC it imposes a smaller penalty on complex models in large samples, which is beneficial when working with large amounts of complex data.

Some links: [wiki](https://en.wikipedia.org/wiki/Hannan%E2%80%93Quinn_information_criterion), [original paper](https://www.jstor.org/stable/2985032), [comparison of different IC for autoregressive models](http://www3.stat.sinica.edu.tw/statistica/oldpdf/A3n214.pdf), [model selection techniques overview](https://arxiv.org/pdf/1810.09583.pdf)

### Describe your proposed solution

Adding two lines of code to existing implementation:
            
`elif self.criterion == "hqic":`
    `    criterion_factor = 2*log(log(n_samples))`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

It shouldn't be too costly to maintain and will provide valid alternatives to AIC and BIC.

COMMENT:
Quick tests with toy regression datasets
Default `cv=5` folds, random seed locked at 1, standard scaler before linear model, `friedman_n` are Friedman problems with `noise=1.0`, scores are R2

|           |       aic |       bic |      hqic |
|:------------|----------:|----------:|----------:|
| diabetes    | 0.479083  | **0.480699**  | 0.479004  |
| boston      | **0.390471**  | 0.386553  | **0.390471**  |
| california  | 0.553031  | **0.553248**  | **0.553248**  |
| friedman1   | 0.586118  | 0.589779  | **0.59488**   |
| friedman2   | **0.808227**  | **0.808227**  | **0.808227**  |
| friedman3   | **0.275709**  | **0.275709**  | **0.275709**  |
| friedman1_n | 0.489092  | **0.498355**  | 0.490848  |
| friedman2_n | **0.808081**  | **0.808081**  | **0.808081**  |
| friedman3_n | 0.0306294 | 0.0145536 | **0.0433222** |

Labels: New Feature, module:linear_model, Needs Decision - Include Feature
Add Hannan-Quinn IC to LassoLarsIC

### Describe the workflow you want to enable

Hannan–Quinn information criterion was originally described to be an alternative model selection criterion for autoregressive models, but it could be used in other cases too. Unlike AIC it also penalizes model's number of parameters, but unlike BIC it imposes a smaller penalty on complex models in large samples, which is beneficial when working with large amounts of complex data.

Some links: [wiki](https://en.wikipedia.org/wiki/Hannan%E2%80%93Quinn_information_criterion), [original paper](https://www.jstor.org/stable/2985032), [comparison of different IC for autoregressive models](http://www3.stat.sinica.edu.tw/statistica/oldpdf/A3n214.pdf), [model selection techniques overview](https://arxiv.org/pdf/1810.09583.pdf)

### Describe your proposed solution

Adding two lines of code to existing implementation:
            
`elif self.criterion == "hqic":`
    `    criterion_factor = 2*log(log(n_samples))`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

It shouldn't be too costly to maintain and will provide valid alternatives to AIC and BIC.

COMMENT:
While the paper that the method was introduced in has enough citations per usual consideration criteria, my only worry is whether it is used commonly enough to warrant addition. In particular, it seems very similar to AIC/BIC and often cited as an "alternative". If it has enough usage in the field as a primary statistic, then it would make sense to include, but otherwise I'm not sure it would be worth it.

I'm not super familiar with it though so I would look for the opinions of those better versed.

Thank you for your patience!

Labels: New Feature, module:linear_model, Needs Decision - Include Feature