Talip7/scikit-learn-issues-multilabel huggingface dataset:
AttributeError: 'LogisticRegression' object has no attribute 'feature_names_in_'

### Describe the bug

I created two model successfully (one being Decision Tree, and the other Logistic Regression). Those were exported as pickle `.pkl` files. Because I used one-hot encoding in both models and I have a lot of categories I needed to get name of features. But weird thing is that I use `feature_names_in_` for both of them and for some reason I get this error  `AttributeError: 'LogisticRegression' object has no attribute 'feature_names_in_'` . On the other hand this works fine for Decision Tree. I also checked the documentation for [Logistic Regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) to double check if this even exists and it does. Also, I have checked version of `sklearn` by running `sklearn.__version__` and indeed I have confirmed latest `1.3.0` version. 

### Steps/Code to Reproduce

```
model = pickle.load(open(model_path, "rb"))
print(f"Version: {sklearn.__version__}")
ordered_columns = model.feature_names_in_
```

### Expected Results

No error is thrown

### Actual Results

```
    prediction = create_prediction(list(model_input_items), "./path_to_model.pkl")
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 create_prediction
    ordered_columns = model.feature_names_in_
                      ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'LogisticRegression' object has no attribute 'feature_names_in_'
```

### Versions

```shell
Version: 1.3.0
```

COMMENT:
The `feature_names_in` attribute is initialized during the `LogisticRegression.fit(X, y)` method call, when the `X` dataset contained string-like column names.

There are two reasons why this attribute might be missing:

1. The model was trained using Scikit-Learn version that is less than 1.0.
2. The `X` dataset did not provide column names (for example, was a Numpy array not a Pandas' DataFrame).

> I have checked version of sklearn by running sklearn.__version__ and indeed I have confirmed latest 1.3.0 version.

What matters is the Scikit-Learn version that dumped this pickle file, not the one that attempts to load it.

It is likely the case that your DecisionTree(Classifier) model was dumped by Scikit-Learn 1.0+, but your LogisticRegression model was dumped by Scikit-Learn 0.24 (or some other pre-1.0 version).

The model object should store the training version information as the `(BaseEstimator.)_sklearn_version` attribute:

```python
model = pickle.load(open(model_path, "rb"))
# THIS - trainer's version
print(model._sklearn_version)
# NOT THIS - it's the loaders' version
#print(f"Version: {sklearn.__version__}")
```

Labels: Bug, Needs Reproducible Code
AttributeError: 'LogisticRegression' object has no attribute 'feature_names_in_'

### Describe the bug

I created two model successfully (one being Decision Tree, and the other Logistic Regression). Those were exported as pickle `.pkl` files. Because I used one-hot encoding in both models and I have a lot of categories I needed to get name of features. But weird thing is that I use `feature_names_in_` for both of them and for some reason I get this error  `AttributeError: 'LogisticRegression' object has no attribute 'feature_names_in_'` . On the other hand this works fine for Decision Tree. I also checked the documentation for [Logistic Regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) to double check if this even exists and it does. Also, I have checked version of `sklearn` by running `sklearn.__version__` and indeed I have confirmed latest `1.3.0` version. 

### Steps/Code to Reproduce

```
model = pickle.load(open(model_path, "rb"))
print(f"Version: {sklearn.__version__}")
ordered_columns = model.feature_names_in_
```

### Expected Results

No error is thrown

### Actual Results

```
    prediction = create_prediction(list(model_input_items), "./path_to_model.pkl")
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 create_prediction
    ordered_columns = model.feature_names_in_
                      ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'LogisticRegression' object has no attribute 'feature_names_in_'
```

### Versions

```shell
Version: 1.3.0
```

COMMENT:
In scikit-learn, the feature_names_in_ attribute is not a standard attribute of the LogisticRegression class. Instead, it's typically associated with tree-based models like Decision Trees or Random Forests.

For logistic regression, you typically don't have a direct attribute like feature_names_in_ to access the feature names. If you need the feature names for logistic regression, you should store them separately in your code before training the model, and then use that stored information when needed.

Labels: Bug, Needs Reproducible Code
AttributeError: 'LogisticRegression' object has no attribute 'feature_names_in_'

### Describe the bug

I created two model successfully (one being Decision Tree, and the other Logistic Regression). Those were exported as pickle `.pkl` files. Because I used one-hot encoding in both models and I have a lot of categories I needed to get name of features. But weird thing is that I use `feature_names_in_` for both of them and for some reason I get this error  `AttributeError: 'LogisticRegression' object has no attribute 'feature_names_in_'` . On the other hand this works fine for Decision Tree. I also checked the documentation for [Logistic Regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) to double check if this even exists and it does. Also, I have checked version of `sklearn` by running `sklearn.__version__` and indeed I have confirmed latest `1.3.0` version. 

### Steps/Code to Reproduce

```
model = pickle.load(open(model_path, "rb"))
print(f"Version: {sklearn.__version__}")
ordered_columns = model.feature_names_in_
```

### Expected Results

No error is thrown

### Actual Results

```
    prediction = create_prediction(list(model_input_items), "./path_to_model.pkl")
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 create_prediction
    ordered_columns = model.feature_names_in_
                      ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'LogisticRegression' object has no attribute 'feature_names_in_'
```

### Versions

```shell
Version: 1.3.0
```

COMMENT:
> In scikit-learn, the feature_names_in_ attribute is not a standard attribute of the LogisticRegression class. Instead, it's typically associated with tree-based models like Decision Trees or Random Forests.

@Shekharmeena28 that's not true. If you pass a pandas dataframe, feature names are stored. Please avoid posting wrong information if you're not sure about the answer.

Closing since no reproducible is provided. Happy to reopen otherwise.

Labels: Bug, Needs Reproducible Code
Add precompute in MultiTask Linear Models

### Describe the workflow you want to enable

Adding a '**precompute**' parameter into constructor of **MultiTaskElasticNet**, **MultiTaskLasso**, **MultiTaskElasticNetCV**, and **MultiTaskLassoCV** in **linear_model/_coordinate_descent.py**
This parameter is only present in SingleTask models(**ElasticNet**, **Lasso**, **LinearModelCV**, **LassoCV**, **ElasticNetCV**), and has shown significant improvements in training speed.

### Describe your proposed solution

In **linear_model/_coordinate_descent.py**, a function **cd_fast.enet_coordinate_descent_multi_task()** in **enet_path()** will be executed when running these MultiTask models. However, this function does not have a precompute version in the **cd_fast.pyx**.

So the solution would be implementing a new function within **linear_model/_cd_fast.pyx**.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Is this because it's complicated or not feasible in mathematics?

COMMENT:
The coordinate descent code is not my strong suit.

@mathurinm @agramfort I assume that the loss for an univariate `y` could be written as well for a multivariate `Y`, something like:

```
(1/2) * W^T X^T X W - (X Y)^T W + ...
```

but I have actually the solver would be easy to write or what type of speed-up to expect.

Would you have some insights regarding this?

Labels: New Feature
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
Thanks. We should be able to observe the regression in the continuous benchmark: https://scikit-learn.org/scikit-learn-benchmarks/

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
Actually we don't have any results uploaded since July 12. Maybe @jeremiedbb can help :)

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
Also we don't use Cython 3 in the main branch yet as far as I know. We only test Cython 3 in the scipy-dev build which is not benchmarked.

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
We updated recently the lock file (during EuroSciPy sprint): https://github.com/scikit-learn/scikit-learn/pull/27024/files and thus we are now using Cython 3.0.

~~We could have a wild check to see if we took an overall slowdown on all test on the CI.~~

I checked and there is too much variation to resort to the test suite.

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
I missed that. Thanks for the heads up.

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
Actually since [there's no upper version pin for Cython](https://github.com/scikit-learn/scikit-learn/blob/main/pyproject.toml#L6), we might have been using Cython 3 since its release.

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
Hm,
I just looked at your asvs and you don't seem to distinguish between Cython 0.29 and Cython 3.0 in yours

See ours for reference https://asv-runner.github.io/asv-collection/pandas/#frame_ctor.FromDicts.time_nested_dict
(In the left sidebar, you can choose between Cython 0.29 and Cython 3.0 while your benchmarks only seem to filter by python and pandas version)

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
I think we need to pin `Cython>=0.29.33<3` except for nightly builds and regenerate locks like before we make sure there's no performance regression with Cython 3.

What do you think? I can't look into that now, but could someone else do? :pray:

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
@jeremiedbb is actually looking at the ASV issue and will run before and after https://github.com/scikit-learn/scikit-learn/pull/27024 that changed the lock files introducing Cython 3.0

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
Actually the use of cython 3 in the benchmarks is not linked to the update of the lock files. The lock files are only for the CI while the asv benchmarks run against the latest version available.
All runs before mid july when the bench stopped being uploaded were run with cython 0.29. The last one that I manually ran yesterday uses cython 3.

From a quick look I can see some severe regressions:
- Random forests: https://scikit-learn.org/scikit-learn-benchmarks/#ensemble.RandomForestClassifierBenchmark.time_fit?python=3.11
- Hist gradient boosting: https://scikit-learn.org/scikit-learn-benchmarks/#ensemble.HistGradientBoostingClassifierBenchmark.time_fit?python=3.11
- KNN: https://scikit-learn.org/scikit-learn-benchmarks/#neighbors.KNeighborsClassifierBenchmark.time_predict?python=3.11

We need more run to check that it does not come from a one time issue with the machine running the benchmarks. Additionnal checks are also needed to make sure cython 3 is the reason of these regressions since the benchmarks were not uploaded for a long time.

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
I added additional runs on asv and the slowdown looks persistent https://scikit-learn.org/scikit-learn-benchmarks/#/.

I could reproduce for the RandomForest locally on the last commit of main, just changing the version of cython. Although the slowdown is lower on my laptop (still x2).
I couldn't reproduce for HGBT. It might be because my laptop has only 2 cores while the machine running the benchs has 8.

It makes me quite confident that cython 3 is the reason of these slowdowns.

I'm also convinced that we should pin the versions of the dependencies in the asv conf to be able to distinguish between regression coming from our code and regression coming from the dependencies. I'm going to open a PR for that.

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
I'll take a look at profiling a couple of the slowdowns. Hopefully I can get a sense of what actual changes may be causing this.

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
@da-woods have you seen other reports of these slowdowns? I'm wondering if there are any hints as to what may be happening

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
General comments - haven't looked into your specific regressions:

The most common issue I've seen is due to the switch from function being implicitly `noexcept` to being able to propagate exceptions by default. Specifically for functions returning `void` this means that there will *always* be an exception check. In a nogil (often parallel) block this means that  they always have to acquire the GIL.

This also applies to functions returning a fused type for slightly more complicated reasons (but the same basic idea).

I'm working on a) a fix for the fused return type and b) better diagnostics of this issue (i.e. to point out places where it might be an issue). Hopefully those will make it into the next release.

Hopefully that gives you a clue where to start looking. If you want I can try to find time to investigate myself (but you'll probably need to give me idiot-proof instructions on how to run the benchmark)

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
I think I forgot to say how to fix it (if this is actually the issue). The choices are:
* Declare the functions causing the issue as `noexcept`,
* Use the `legacy_implicit_noexcept` compiler directive - I don't really recommend because it's a catch all, but I'll give you a quick diagnostic about if this is the issue,
* Use a dummy `int` return type (and just return 0 on success), this lets Cython use `-1` as an error return type, so you get the benefit of the improved exception propagation with much less performance cost.

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
I profiled the RandomForestClassifier with py-spy and it shows that with cython 3 there's a lot of overhead coming from pthread locks that were not there before.
![Screenshot_cy3](https://github.com/scikit-learn/scikit-learn/assets/34657725/acd2fea8-2ef5-477f-ba31-0df5b35d3029)
Everything circled in blue is related to pthreads locks. I can't figure out what causes that from looking at the functions where it happens, e.g. https://github.com/scikit-learn/scikit-learn/blob/a05eb6bbe30bc40904274c87afb4ac93bcd60168/sklearn/tree/_criterion.pyx#L487-L554

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
From the graph you show it definitely looks like a "waiting on the GIL" issue. Although it isn't obvious quite what's causing it in that function. There's also an issue that's been reported about acquiring the GIL for memoryview reference counting at the end of the function, so it's possibly that. That issue would need a fix on our side. It might go away if you used `sample_indices` and `sample_weights` from the class rather than assigning them to local variables. But I imagine that's done for good reason.

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
#27266 has been opened to have scikit-learn depend on Cython < 3.

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
Would it be possible to remeasure RandomForestClassifier with https://github.com/cython/cython/pull/5678? My semi-informed suspicion is that this might well fix it. (Or to let me know how to run the benchmark and I can measure it)

Labels: Performance, Needs Benchmarks, cython
DIS: Cython 3 perf regressions

It looks like scikit-learn hasn't updated to Cython 3 yet, but when you do, can you keep an eye out for performance regressions?

On pandas, we updated to Cython 3 but then reverted, since we had issues with some of our benchmarks regressing, and it would be nice to know if this is a bigger problem in Cython.

COMMENT:
FYI, it seems like CircleCI timeout issues are due (for the most part at least) to Cython 3. In particular `examples/ensemble/plot_forest_hist_grad_boosting_comparison.py` now takes ~8 minutes rather than 1 minute before we updated the lock files to Cython 3 and it uses a random forest.

Labels: Performance, Needs Benchmarks, cython