Talip7/scikit-learn-issues-multilabel huggingface dataset:
Exposing submodules directly from main sklearn module instead of recursively importing each submodule separately

### Describe the workflow you want to enable

The problem that I'm facing is the same as the issue described in this [stackoverflow](https://stackoverflow.com/questions/46572475/module-sklearn-has-no-attribute-cross-validation#comment127202377_46572475) page. Basically sklearn lacks the ability to allow users flexibility in how they would like to import submodules.

For instance, if someone wants to use `x,y,z,...,etc` submodule form sklearn they would have to do the following for each of them:
```
from sklearn.x import x
from sklearn.y import y
from sklearn.z import z
```

This not only increases the line of codes in a codebase but also makes core readability more difficult to follow, especially in large code bases. 

### Describe your proposed solution

The best thing would be to enable the same functionality as numpy.
For instance, in numpy all we have to do is `import numpy as np` and all the magic is in the `np.` namespace.
I would really like to be able to do the same thing with sklearn, e.g., `import sklearn as sk` and then calling the different submodules as needed, e.g., `sk.metrics.auccuracy` or `sk.model_selection.cross_validate`, etc.

At least this is a coherent way of reading the code and understanding much easier where each submodule came from.

I really don't know why sklearn has avoided this kind of exposure of submodules since all other major libraries do it, e.g., `import pandas as pd`?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
There is overhead in loading all the submodules. scikit-learn is more similar to SciPy, where most functionality are in submodules. For example, the following does not work in SciPy:

```python
import scipy as sp

sp.linalg.eig(...)
```

The issue and a lazy loading solution are described in [Scientific Python Enhancement Proposal 1](https://scientific-python.org/specs/spec-0001/).

Labels: New Feature, Needs Decision - Include Feature
Exposing submodules directly from main sklearn module instead of recursively importing each submodule separately

### Describe the workflow you want to enable

The problem that I'm facing is the same as the issue described in this [stackoverflow](https://stackoverflow.com/questions/46572475/module-sklearn-has-no-attribute-cross-validation#comment127202377_46572475) page. Basically sklearn lacks the ability to allow users flexibility in how they would like to import submodules.

For instance, if someone wants to use `x,y,z,...,etc` submodule form sklearn they would have to do the following for each of them:
```
from sklearn.x import x
from sklearn.y import y
from sklearn.z import z
```

This not only increases the line of codes in a codebase but also makes core readability more difficult to follow, especially in large code bases. 

### Describe your proposed solution

The best thing would be to enable the same functionality as numpy.
For instance, in numpy all we have to do is `import numpy as np` and all the magic is in the `np.` namespace.
I would really like to be able to do the same thing with sklearn, e.g., `import sklearn as sk` and then calling the different submodules as needed, e.g., `sk.metrics.auccuracy` or `sk.model_selection.cross_validate`, etc.

At least this is a coherent way of reading the code and understanding much easier where each submodule came from.

I really don't know why sklearn has avoided this kind of exposure of submodules since all other major libraries do it, e.g., `import pandas as pd`?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thanks for the reply and pointing to a solution. Since lazy loading avoids the overhead you mentioned wouldn't it make more sense to implement this in `sklearn` given that the user base is quite broad and cannot expect everyone to implement lazy loading by modifying `__init__.py` files?

Labels: New Feature, Needs Decision - Include Feature
Exposing submodules directly from main sklearn module instead of recursively importing each submodule separately

### Describe the workflow you want to enable

The problem that I'm facing is the same as the issue described in this [stackoverflow](https://stackoverflow.com/questions/46572475/module-sklearn-has-no-attribute-cross-validation#comment127202377_46572475) page. Basically sklearn lacks the ability to allow users flexibility in how they would like to import submodules.

For instance, if someone wants to use `x,y,z,...,etc` submodule form sklearn they would have to do the following for each of them:
```
from sklearn.x import x
from sklearn.y import y
from sklearn.z import z
```

This not only increases the line of codes in a codebase but also makes core readability more difficult to follow, especially in large code bases. 

### Describe your proposed solution

The best thing would be to enable the same functionality as numpy.
For instance, in numpy all we have to do is `import numpy as np` and all the magic is in the `np.` namespace.
I would really like to be able to do the same thing with sklearn, e.g., `import sklearn as sk` and then calling the different submodules as needed, e.g., `sk.metrics.auccuracy` or `sk.model_selection.cross_validate`, etc.

At least this is a coherent way of reading the code and understanding much easier where each submodule came from.

I really don't know why sklearn has avoided this kind of exposure of submodules since all other major libraries do it, e.g., `import pandas as pd`?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
It's all technically possible. It would be similar to how `scikit-image` is currently lazy loading their [filters module](https://github.com/scikit-image/scikit-image/blob/4a5186a53b89ac764b822704b12c0bcd3c10606f/skimage/filters/__init__.py#L1-L5). They vendor the `lazy-import` library and use it in `__init__.py`.

For scikit-learn, the work is mostly developer documentation so that contributors understand how to work with the lazy loading system. We also need tests to make sure that all public functions are registered with the lazy loading system.

Before all that, scikit-learn maintainers need to decide if we want to adopt [Scientific Python Enhancement Proposal 1](https://scientific-python.org/specs/spec-0001/) and that the maintenance cost of lazy loading is worth it.

Labels: New Feature, Needs Decision - Include Feature
Exposing submodules directly from main sklearn module instead of recursively importing each submodule separately

### Describe the workflow you want to enable

The problem that I'm facing is the same as the issue described in this [stackoverflow](https://stackoverflow.com/questions/46572475/module-sklearn-has-no-attribute-cross-validation#comment127202377_46572475) page. Basically sklearn lacks the ability to allow users flexibility in how they would like to import submodules.

For instance, if someone wants to use `x,y,z,...,etc` submodule form sklearn they would have to do the following for each of them:
```
from sklearn.x import x
from sklearn.y import y
from sklearn.z import z
```

This not only increases the line of codes in a codebase but also makes core readability more difficult to follow, especially in large code bases. 

### Describe your proposed solution

The best thing would be to enable the same functionality as numpy.
For instance, in numpy all we have to do is `import numpy as np` and all the magic is in the `np.` namespace.
I would really like to be able to do the same thing with sklearn, e.g., `import sklearn as sk` and then calling the different submodules as needed, e.g., `sk.metrics.auccuracy` or `sk.model_selection.cross_validate`, etc.

At least this is a coherent way of reading the code and understanding much easier where each submodule came from.

I really don't know why sklearn has avoided this kind of exposure of submodules since all other major libraries do it, e.g., `import pandas as pd`?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Given that other major players such as scipy are moving this way, user expectation of this behaviour will only increase; I'm sure the current behaviour surprises many new python users. I'm +1 for this

Labels: New Feature, Needs Decision - Include Feature
Exposing submodules directly from main sklearn module instead of recursively importing each submodule separately

### Describe the workflow you want to enable

The problem that I'm facing is the same as the issue described in this [stackoverflow](https://stackoverflow.com/questions/46572475/module-sklearn-has-no-attribute-cross-validation#comment127202377_46572475) page. Basically sklearn lacks the ability to allow users flexibility in how they would like to import submodules.

For instance, if someone wants to use `x,y,z,...,etc` submodule form sklearn they would have to do the following for each of them:
```
from sklearn.x import x
from sklearn.y import y
from sklearn.z import z
```

This not only increases the line of codes in a codebase but also makes core readability more difficult to follow, especially in large code bases. 

### Describe your proposed solution

The best thing would be to enable the same functionality as numpy.
For instance, in numpy all we have to do is `import numpy as np` and all the magic is in the `np.` namespace.
I would really like to be able to do the same thing with sklearn, e.g., `import sklearn as sk` and then calling the different submodules as needed, e.g., `sk.metrics.auccuracy` or `sk.model_selection.cross_validate`, etc.

At least this is a coherent way of reading the code and understanding much easier where each submodule came from.

I really don't know why sklearn has avoided this kind of exposure of submodules since all other major libraries do it, e.g., `import pandas as pd`?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
We discussed this issue during the developer meeting on April 25.
The conclusion was that we need to experiment before making a decision.
Notably, we should investigate the following:

- is there any memory and runtime overhead in the multiprocessing setting?

Labels: New Feature, Needs Decision - Include Feature
Consider LSMR instead of LSQR

#### Proposition
Currently, we use the LSQR solver for (possibly sparse) linear equation / least squares problems in
- `LinearDiscriminantAnalysis`
- `LinearRegression`
- `Ridge`

Consider to change to LSMR because according to [1] and [2] LSMR might be able to terminate earlier and be faster. See in particular Fig 1 of [2] which compares LSMR and LSQR on CUTEst and UFL test sets of 921 eligible problems with no preconditioning.

If we decide to do so, how do we implement the change?
- Just replace `lsqr` with `lsmr` and document it; or
- Add new solver option `lsmr`.


#### References
[1] https://stanford.edu/group/SOL/software/lsmr/

[2] Gould, N.I., & Scott, J.A. (2017). The State-of-the-Art of Preconditioners for Sparse Linear Least-Squares Problems. ACM Transactions on Mathematical Software (TOMS), 43, 1 - 35.
   http://centaur.reading.ac.uk/70342/1/ls_study_toms.pdf

COMMENT:
> If we decide to do so, how do we implement the change?

I think that we should better use a deprecation cycle: introduce the new solver and announce a change of default in 2 versions.

I think that we should still have a couple of benchmarks when introducing the solver to be sure that the scipy implementation is in line with the literature.

Labels: module:linear_model, module:discriminant_analysis
Different results depending on labels using linear_svc and adaboost

Hi all, 

I'm having different results with linear_svc and adaboost depending on the labels `0/1 or 1/-1`. Below I've provided github repo with a notebook example using part of the[ CoronaHack -Chest X-Ray-Datase](https://www.kaggle.com/datasets/praveengovi/coronahack-chest-xraydataset?resource=download) data. 

I have tried with other datasets and the behavior is similar.  I would expect that the labels shouldn't matter, the notebook shows a simple example. I couldn't find any error there. 

Any help would be appreciated, thanks!

### Steps/Code to Reproduce

Repo to reproduce results: https://github.com/victor-onofre/Labels_testing

## Expected Results

### **Results labels 0/1**

_**_linear_svc:_**_ 

- Linear SVC precision: 0.899581589958159

- Linear SVC recall score: 0.7992565055762082

-  Linear SVC F1-score: 0.8464566929133858

- Running time: 0.27 seconds (0.0 minutes)

**_adaboost:_**

- Number of Decision Tree Classifiers: 50
- AdaBoost precision: 0.8524590163934426
- AdaBoost recall score: 0.7732342007434945
- AdaBoost F1-score: 0.810916179337232

- Running time: 1.201 seconds (0.0 minutes)

## Actual Results

### **Results labels 1/-1**

_**_linear_svc:_**_ 
- Linear SVC precision: 0.9340659340659341
- Linear SVC recall score: 0.9695817490494296
- Linear SVC F1-score: 0.9514925373134328

- Running time: 0.27 seconds (0.0 minutes)

**_adaboost:_**

- AdaBoost precision: 0.9280487804878049
- AdaBoost recall score: 0.9645120405576679
- AdaBoost F1-score: 0.9459291485394654

- Running time: 0.969 seconds (0.0 minutes)



### Versions

```shell
System:
    python: 3.7.13 (default, Apr 20 2022, 19:06:16)  [GCC 10.2.1 20210110]
executable: /usr/local/bin/python3
   machine: Linux-5.10.76-linuxkit-x86_64-with-debian-11.3

Python dependencies:
          pip: 22.0.4
   setuptools: 57.5.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.7.1
       Cython: None
       pandas: 1.3.5
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
At a glance, I suspect it's how `precision_score` and the other metrics uses `pos_label=1` by default to define the positive class. The values of these metrics depends on which class is considered positive. There are two options to fix this:

1. Use -1/1 encoding so that class 1 is always the positive class for both encodings.
2. When using 1/-1 encoding set `pos_label=-1` in `precision_score` and the other metrics.

Labels: Question
Different results depending on labels using linear_svc and adaboost

Hi all, 

I'm having different results with linear_svc and adaboost depending on the labels `0/1 or 1/-1`. Below I've provided github repo with a notebook example using part of the[ CoronaHack -Chest X-Ray-Datase](https://www.kaggle.com/datasets/praveengovi/coronahack-chest-xraydataset?resource=download) data. 

I have tried with other datasets and the behavior is similar.  I would expect that the labels shouldn't matter, the notebook shows a simple example. I couldn't find any error there. 

Any help would be appreciated, thanks!

### Steps/Code to Reproduce

Repo to reproduce results: https://github.com/victor-onofre/Labels_testing

## Expected Results

### **Results labels 0/1**

_**_linear_svc:_**_ 

- Linear SVC precision: 0.899581589958159

- Linear SVC recall score: 0.7992565055762082

-  Linear SVC F1-score: 0.8464566929133858

- Running time: 0.27 seconds (0.0 minutes)

**_adaboost:_**

- Number of Decision Tree Classifiers: 50
- AdaBoost precision: 0.8524590163934426
- AdaBoost recall score: 0.7732342007434945
- AdaBoost F1-score: 0.810916179337232

- Running time: 1.201 seconds (0.0 minutes)

## Actual Results

### **Results labels 1/-1**

_**_linear_svc:_**_ 
- Linear SVC precision: 0.9340659340659341
- Linear SVC recall score: 0.9695817490494296
- Linear SVC F1-score: 0.9514925373134328

- Running time: 0.27 seconds (0.0 minutes)

**_adaboost:_**

- AdaBoost precision: 0.9280487804878049
- AdaBoost recall score: 0.9645120405576679
- AdaBoost F1-score: 0.9459291485394654

- Running time: 0.969 seconds (0.0 minutes)



### Versions

```shell
System:
    python: 3.7.13 (default, Apr 20 2022, 19:06:16)  [GCC 10.2.1 20210110]
executable: /usr/local/bin/python3
   machine: Linux-5.10.76-linuxkit-x86_64-with-debian-11.3

Python dependencies:
          pip: 22.0.4
   setuptools: 57.5.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.7.1
       Cython: None
       pandas: 1.3.5
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
That was the error. Thank you so much @thomasjpfan!

Labels: Question
The azure CI times out more and more often

- The 2 windows jobs timeout (60min) more often than they don't these days. When they manage to complete, they take ~55min. I recall that they used to take "only" ~40min not so long ago.

- The 2 macos jobs don't timeout but usually take ~50min while they used to take ~30min.

- Even the linux jobs take more time than before.

COMMENT:
for reference, locally (linux, 4 cores) it runs in less than 17min, without xdist.
EDIT: locally it only runs ~25000 tests while it's ~90000 tests in the CI !

Labels: Build / CI
The azure CI times out more and more often

- The 2 windows jobs timeout (60min) more often than they don't these days. When they manage to complete, they take ~55min. I recall that they used to take "only" ~40min not so long ago.

- The 2 macos jobs don't timeout but usually take ~50min while they used to take ~30min.

- Even the linux jobs take more time than before.

COMMENT:
So it's probably due to this addition:
```py
if [[ "$SELECTED_TESTS" != "" ]]; then
    TEST_CMD="$TEST_CMD -k $SELECTED_TESTS"

    # Override to make selected tests run on all random seeds
    export SKLEARN_TESTS_GLOBAL_RANDOM_SEED="all"
fi
```
the CI shows the `-k` in the pytest invocation command meaning we are entering the test

Labels: Build / CI
Investigate SAG/SAGA solver

#### Description
The newly introduced tight tests for `Ridge` in #22910 together with the random seed fixture in #22749 revealed some shortcomings of the sag and saga solver:

1. ~~It shows some random behavior even with fixed random seed.~~
2. The `tol` needs to be set much smaller to receive comparable results with the other solvers of `Ridge`.

Ideally, the cause for both issues can be identified and fixed.

#### Some links for context
https://github.com/scikit-learn/scikit-learn/issues/23014
https://github.com/scikit-learn/scikit-learn/pull/23017
https://github.com/scikit-learn/scikit-learn/pull/23026
https://github.com/scikit-learn/scikit-learn/pull/23152
#23177

COMMENT:
> It shows some random behavior even with fixed random seed.

Actually the randomness will probably be fixed by https://github.com/scikit-learn/scikit-learn/pull/23181
It was probably not randomness from the solver but from the CI runs. The fact that when it failed it was always with sag and saga still shows that something is not going as expected.

Labels: Bug, Moderate, help wanted, module:linear_model
Investigate SAG/SAGA solver

#### Description
The newly introduced tight tests for `Ridge` in #22910 together with the random seed fixture in #22749 revealed some shortcomings of the sag and saga solver:

1. ~~It shows some random behavior even with fixed random seed.~~
2. The `tol` needs to be set much smaller to receive comparable results with the other solvers of `Ridge`.

Ideally, the cause for both issues can be identified and fixed.

#### Some links for context
https://github.com/scikit-learn/scikit-learn/issues/23014
https://github.com/scikit-learn/scikit-learn/pull/23017
https://github.com/scikit-learn/scikit-learn/pull/23026
https://github.com/scikit-learn/scikit-learn/pull/23152
#23177

COMMENT:
If it can reach the same results with a small values of `tol`, it does not seem like a bug to me.

Are there any example where the solvers do not converge to the same solution even with a small `tol`?

Labels: Bug, Moderate, help wanted, module:linear_model
Investigate SAG/SAGA solver

#### Description
The newly introduced tight tests for `Ridge` in #22910 together with the random seed fixture in #22749 revealed some shortcomings of the sag and saga solver:

1. ~~It shows some random behavior even with fixed random seed.~~
2. The `tol` needs to be set much smaller to receive comparable results with the other solvers of `Ridge`.

Ideally, the cause for both issues can be identified and fixed.

#### Some links for context
https://github.com/scikit-learn/scikit-learn/issues/23014
https://github.com/scikit-learn/scikit-learn/pull/23017
https://github.com/scikit-learn/scikit-learn/pull/23026
https://github.com/scikit-learn/scikit-learn/pull/23152
#23177

COMMENT:
The point is that sag/saga often needs orders of magnitude smaller `tol`. In `test_ridge.py` it's `1e-15` vs `1e-10`. We could (and do) live with that. Maybe there is room to improve on that.

Labels: Bug, Moderate, help wanted, module:linear_model
Investigate SAG/SAGA solver

#### Description
The newly introduced tight tests for `Ridge` in #22910 together with the random seed fixture in #22749 revealed some shortcomings of the sag and saga solver:

1. ~~It shows some random behavior even with fixed random seed.~~
2. The `tol` needs to be set much smaller to receive comparable results with the other solvers of `Ridge`.

Ideally, the cause for both issues can be identified and fixed.

#### Some links for context
https://github.com/scikit-learn/scikit-learn/issues/23014
https://github.com/scikit-learn/scikit-learn/pull/23017
https://github.com/scikit-learn/scikit-learn/pull/23026
https://github.com/scikit-learn/scikit-learn/pull/23152
#23177

COMMENT:
Coming back to this issue, I think it would be interesting to find a way to normalize the `tol` criterion between all ridge solvers including sag/saga so that they approximately mean the same thing.

Labels: Bug, Moderate, help wanted, module:linear_model
Investigate SAG/SAGA solver

#### Description
The newly introduced tight tests for `Ridge` in #22910 together with the random seed fixture in #22749 revealed some shortcomings of the sag and saga solver:

1. ~~It shows some random behavior even with fixed random seed.~~
2. The `tol` needs to be set much smaller to receive comparable results with the other solvers of `Ridge`.

Ideally, the cause for both issues can be identified and fixed.

#### Some links for context
https://github.com/scikit-learn/scikit-learn/issues/23014
https://github.com/scikit-learn/scikit-learn/pull/23017
https://github.com/scikit-learn/scikit-learn/pull/23026
https://github.com/scikit-learn/scikit-learn/pull/23152
#23177

COMMENT:
could this be connected to #24679 ?

Labels: Bug, Moderate, help wanted, module:linear_model
Investigate SAG/SAGA solver

#### Description
The newly introduced tight tests for `Ridge` in #22910 together with the random seed fixture in #22749 revealed some shortcomings of the sag and saga solver:

1. ~~It shows some random behavior even with fixed random seed.~~
2. The `tol` needs to be set much smaller to receive comparable results with the other solvers of `Ridge`.

Ideally, the cause for both issues can be identified and fixed.

#### Some links for context
https://github.com/scikit-learn/scikit-learn/issues/23014
https://github.com/scikit-learn/scikit-learn/pull/23017
https://github.com/scikit-learn/scikit-learn/pull/23026
https://github.com/scikit-learn/scikit-learn/pull/23152
#23177

COMMENT:
> could this be connected to #24679 ?

No. The linked issue is a potential mistake on the prox operator in SAGA. However, the prox operator is not used in Ridge, because the prox operator is only used for L1 regularization. Therefore, this potential mistake should not affect Ridge convergence.

edit: #24679 was not a mistake.

Labels: Bug, Moderate, help wanted, module:linear_model