Talip7/scikit-learn-issues-multilabel huggingface dataset:
Fix ConvergenceWarning in `plot_gpr_on_structured_data.py` example

This issue is about addressing a `ConvergenceWarning` that occurs when running the `examples/gaussian_process/plot_gpr_on_structured_data.py `example in CI (also when building the documentation locally).

The example creates three plots. The last use case on a classification of DNA sequences throws a `ConvergenceWarning` related to the `baseline_similarity_bounds` defined in a custom kernel when fitting. It seems that the lower bound is pushed resulting in the lack of convergence.

This occurs with the setting `baseline_similarity_bounds=(1e-5, 1))` in the custom kernel.

Even setting `baseline_similarity_bounds=(1e-40, 1)) ` results in the same warning:
```
ConvergenceWarning: The optimal value found for dimension 0 of parameter baseline_similarity is close to the specified lower bound 1e-40. Decreasing the bound and calling fit again may find a better value.
```

Lowering the bound further with `baseline_similarity_bounds=(1e-50, 1)) ` results in a different warning stemming from `lbfgs`:
```
ConvergenceWarning: lbfgs failed to converge (status=2): ABNORMAL: .
Increase the number of iterations (max_iter) or scale the data as shown in: https://scikit-learn.org/stable/modules/preprocessing.html
```

It would be preferable to resolve this so the example can be build without displaying warnings.


While being at the example, other small improvements are welcome (for instance fixing the typo in "use of kernel functions that operates" (the s in operates)).

COMMENT:
Hello @EngineerDanny,

thank your for looking into it!

This is a bit outside of my comfort zone, but I would think the pragmatic solution with setting the param to "fixed" is good enough, when we also add a short note/comment stating that `baseline_similarity_bounds` is usually used to set optimization bounds, but here is set to "fixed" to circumvent a `ConvergenceWarning` specific to this example.

What do you think, @jeremiedbb?

Labels: help wanted, Needs Investigation
Nearest neighbors Gaussian Process

### Describe the workflow you want to enable

Recently I've been working on a Nearest Neighbor Gaussian Process Regressor as described in Datta 2016 [here](https://arxiv.org/abs/1406.7343). This kind of model exists in R, but not in scikit-learn. Nearest Neighbor Gaussian Process Regressor is a simple enhancement over standard GP that allows to use GP on large datasets. It also recently gained interest among the GPytorch package, see e.g. [here](https://arxiv.org/abs/2202.01694).



### Describe your proposed solution

I already have a scikit-learn-like implementation that I could bring to this project. This implementation becomes more convenient (uses less memory and less runtime) than classic Gaussian Process Regressor from a dataset size of approx 10k. It is based on Datta's work, so it's not as the one in the GPytorch package. If anyone deems this model interesting enough, I'm wiling to make a PR.

Having a baseline CPU-base implementation in scikit-learn could also server as a starting point for future GPU-based implementations, which is were this model really shines (e.g. inheriting from scikit-learn class and implementing in GPU the most time consuming operations). As an example, I also have a cupy-based implementation of Datta's NNGP which competes very well against GPytorch VNNGP.

### Describe alternatives you've considered, if relevant

As mentioned above, a version of NNGP is implemented in GPytorch. GPytorch implementation however is not only based on Nearest Neighbors, but also on Variational method. The one from Datta's is simpler being only based on NN and can become competitive with more complex methods VNNGP when using GPUs.

### Additional context

_No response_

COMMENT:
According to google scholar, the paper has 775 citations. However I am not familiar enough with the GP literature to properly assess the usefulness/maintenance tradeoff.

If you already have a working implementation, could you please publish it or link to it (e.g. as a personal repo on github) to get an idea on how complex is the code?

> This implementation becomes more convenient (uses less memory and less runtime) than classic Gaussian Process Regressor from a dataset size of approx 10k.

Have you tried to run benchmarks? On which kinds of datasets / tasks was this most useful to you?

In particular, do you use it on highdimensional or small dimensional data. I have the feeling that using ball-tree or kd-tree indices might make this work fast in low dimensional data (<~ 10 dimensions).

> As an example, I also have a cupy-based implementation of Datta's NNGP which competes very well against GPytorch VNNGP.

Same remark: I would love to see code and numbers to back this claim.

Labels: New Feature, Needs Decision - Include Feature
Nearest neighbors Gaussian Process

### Describe the workflow you want to enable

Recently I've been working on a Nearest Neighbor Gaussian Process Regressor as described in Datta 2016 [here](https://arxiv.org/abs/1406.7343). This kind of model exists in R, but not in scikit-learn. Nearest Neighbor Gaussian Process Regressor is a simple enhancement over standard GP that allows to use GP on large datasets. It also recently gained interest among the GPytorch package, see e.g. [here](https://arxiv.org/abs/2202.01694).



### Describe your proposed solution

I already have a scikit-learn-like implementation that I could bring to this project. This implementation becomes more convenient (uses less memory and less runtime) than classic Gaussian Process Regressor from a dataset size of approx 10k. It is based on Datta's work, so it's not as the one in the GPytorch package. If anyone deems this model interesting enough, I'm wiling to make a PR.

Having a baseline CPU-base implementation in scikit-learn could also server as a starting point for future GPU-based implementations, which is were this model really shines (e.g. inheriting from scikit-learn class and implementing in GPU the most time consuming operations). As an example, I also have a cupy-based implementation of Datta's NNGP which competes very well against GPytorch VNNGP.

### Describe alternatives you've considered, if relevant

As mentioned above, a version of NNGP is implemented in GPytorch. GPytorch implementation however is not only based on Nearest Neighbors, but also on Variational method. The one from Datta's is simpler being only based on NN and can become competitive with more complex methods VNNGP when using GPUs.

### Additional context

_No response_

COMMENT:
> If you already have a working implementation, could you please publish it or link to it (e.g. as a personal repo on github) to get an idea on how complex is the code?

Sure, give me a couple of days to clean it up a bit, add readme, unittest, then I will publish it.

> Have you tried to run benchmarks? On which kinds of datasets / tasks was this most useful to you?
In particular, do you use it on highdimensional or small dimensional data. I have the feeling that using ball-tree or kd-tree indices might make this work fast in low dimensional data (<~ 10 dimensions).

Right observation, I also expect the dimensionality of the problem to affect the speed. My statement (10k observations) was based on the 3D-road dataset, which indeed is low dimensional. I'm now checking on something more high-dimensional, like the elevators dataset. Linking to the question above, I can include few benchmarks in the code I will publish.

>As an example, I also have a cupy-based implementation of Datta's NNGP which competes very well against GPytorch VNNGP.

If relevant, I can publish some code of what above as well. Though, being cupy-based, I thought it was not much relevant in the context of scikit-learn.

Labels: New Feature, Needs Decision - Include Feature
Nearest neighbors Gaussian Process

### Describe the workflow you want to enable

Recently I've been working on a Nearest Neighbor Gaussian Process Regressor as described in Datta 2016 [here](https://arxiv.org/abs/1406.7343). This kind of model exists in R, but not in scikit-learn. Nearest Neighbor Gaussian Process Regressor is a simple enhancement over standard GP that allows to use GP on large datasets. It also recently gained interest among the GPytorch package, see e.g. [here](https://arxiv.org/abs/2202.01694).



### Describe your proposed solution

I already have a scikit-learn-like implementation that I could bring to this project. This implementation becomes more convenient (uses less memory and less runtime) than classic Gaussian Process Regressor from a dataset size of approx 10k. It is based on Datta's work, so it's not as the one in the GPytorch package. If anyone deems this model interesting enough, I'm wiling to make a PR.

Having a baseline CPU-base implementation in scikit-learn could also server as a starting point for future GPU-based implementations, which is were this model really shines (e.g. inheriting from scikit-learn class and implementing in GPU the most time consuming operations). As an example, I also have a cupy-based implementation of Datta's NNGP which competes very well against GPytorch VNNGP.

### Describe alternatives you've considered, if relevant

As mentioned above, a version of NNGP is implemented in GPytorch. GPytorch implementation however is not only based on Nearest Neighbors, but also on Variational method. The one from Datta's is simpler being only based on NN and can become competitive with more complex methods VNNGP when using GPUs.

### Additional context

_No response_

COMMENT:
As requested, I published the model in [this project](https://github.com/stavoltafunzia/nngpr). I have also added some very basic benchmarks.
In the coming few days I will also add the cuda-based implementation.

Labels: New Feature, Needs Decision - Include Feature
Nearest neighbors Gaussian Process

### Describe the workflow you want to enable

Recently I've been working on a Nearest Neighbor Gaussian Process Regressor as described in Datta 2016 [here](https://arxiv.org/abs/1406.7343). This kind of model exists in R, but not in scikit-learn. Nearest Neighbor Gaussian Process Regressor is a simple enhancement over standard GP that allows to use GP on large datasets. It also recently gained interest among the GPytorch package, see e.g. [here](https://arxiv.org/abs/2202.01694).



### Describe your proposed solution

I already have a scikit-learn-like implementation that I could bring to this project. This implementation becomes more convenient (uses less memory and less runtime) than classic Gaussian Process Regressor from a dataset size of approx 10k. It is based on Datta's work, so it's not as the one in the GPytorch package. If anyone deems this model interesting enough, I'm wiling to make a PR.

Having a baseline CPU-base implementation in scikit-learn could also server as a starting point for future GPU-based implementations, which is were this model really shines (e.g. inheriting from scikit-learn class and implementing in GPU the most time consuming operations). As an example, I also have a cupy-based implementation of Datta's NNGP which competes very well against GPytorch VNNGP.

### Describe alternatives you've considered, if relevant

As mentioned above, a version of NNGP is implemented in GPytorch. GPytorch implementation however is not only based on Nearest Neighbors, but also on Variational method. The one from Datta's is simpler being only based on NN and can become competitive with more complex methods VNNGP when using GPUs.

### Additional context

_No response_

COMMENT:
I added the Cuda-based NNGPR in the [same project](https://github.com/stavoltafunzia/nngpr/tree/main) to give an idea of the possible speedup. See, for example, benchmark.ipynb.

Labels: New Feature, Needs Decision - Include Feature
Enable exporting trained models to text files to be able to import later

### Describe the workflow you want to enable

```python
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor

# make data
X,y = fetch_california_housing(return_X_y=True)

# instantiate Random-Forest and fit it
rf_model = RandomForestRegressor(min_samples_leaf=5, random_state=0, n_jobs=-1)
rf_model.fit(X, y)

# export model to a text file, inspired by https://xgboost.readthedocs.io/en/release_3.0.0/python/python_api.html#xgboost.XGBRegressor.save_model
rf_model.save_model("model.json")


#################### in a new python environment
from sklearn.ensemble import RandomForestRegressor
rf_model2 = RandomForestRegressor(min_samples_leaf=5, random_state=0, n_jobs=-1)
rf_model2.load_model("model.json")
```

### Describe your proposed solution

The current recommended way I believe is to export fit (or trained) models is to serialize them using joblib, which depends on python version, joblib version and scikit-learn version too, and I presume this may lead to issues with OS and CPU architecture as well (windows or  GNU Linux and x86 or ARM64).

So, I request a way to export model's trained weights (or other relevant things like bins & trees for RandomForestRegressor) for it be to loaded from any scikit-learn version or python version or operating system. This is just how the big packages like [xgboost](https://xgboost.readthedocs.io/en/release_3.0.0/python/python_api.html#xgboost.XGBRegressor.save_model) and pytorch ([using `state_dict`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.load_state_dict)) and hence transformers,  handle things. 

This would enable to change environments and platforms easily without having to train model for the new package-versions, architecture and OS again or every time an update in them is required.

### Describe alternatives you've considered, if relevant

There is no alternative to everything that scikit-learn offers as of now.

### Additional context

This has been provied above.

COMMENT:
Thank you for your time and your comment.

I came across a table outlining the pros and cons of using skops at https://scikit-learn.org/stable/model_persistence.html. The primary reason I created this github-issue is to address compatibility across different versions.

In the company I work for, we have a few ML models trained using Scikit-learn that are currently in production. These models rarely require updates, so retraining is largely out of scope. However, over time, due to security and maintenance requirements, we need to upgrade the production environment — including the Python and Scikit-learn versions. Ideally, we’d like to upgrade the environment without needing to retrain the models.

Unfortunately, from what I understand, skops does not currently support this use case well and this seems to be out of scope even for scikit-learn.

Labels: New Feature, Needs Decision - Include Feature
Enable exporting trained models to text files to be able to import later

### Describe the workflow you want to enable

```python
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor

# make data
X,y = fetch_california_housing(return_X_y=True)

# instantiate Random-Forest and fit it
rf_model = RandomForestRegressor(min_samples_leaf=5, random_state=0, n_jobs=-1)
rf_model.fit(X, y)

# export model to a text file, inspired by https://xgboost.readthedocs.io/en/release_3.0.0/python/python_api.html#xgboost.XGBRegressor.save_model
rf_model.save_model("model.json")


#################### in a new python environment
from sklearn.ensemble import RandomForestRegressor
rf_model2 = RandomForestRegressor(min_samples_leaf=5, random_state=0, n_jobs=-1)
rf_model2.load_model("model.json")
```

### Describe your proposed solution

The current recommended way I believe is to export fit (or trained) models is to serialize them using joblib, which depends on python version, joblib version and scikit-learn version too, and I presume this may lead to issues with OS and CPU architecture as well (windows or  GNU Linux and x86 or ARM64).

So, I request a way to export model's trained weights (or other relevant things like bins & trees for RandomForestRegressor) for it be to loaded from any scikit-learn version or python version or operating system. This is just how the big packages like [xgboost](https://xgboost.readthedocs.io/en/release_3.0.0/python/python_api.html#xgboost.XGBRegressor.save_model) and pytorch ([using `state_dict`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.load_state_dict)) and hence transformers,  handle things. 

This would enable to change environments and platforms easily without having to train model for the new package-versions, architecture and OS again or every time an update in them is required.

### Describe alternatives you've considered, if relevant

There is no alternative to everything that scikit-learn offers as of now.

### Additional context

This has been provied above.

COMMENT:
I understand the rationale behind internal changes in scikit-learn over time. However, I’m curious — why are these changes significant enough that models exported in one version cannot be reliably read in future versions?

May I ask why long-term backward compatibility is considered out of scope? For instance, PyTorch — which arguably has a more complex architecture — manages to maintain compatibility of serialized models across versions to a great extent. I’m wondering what the key differences or constraints are in scikit-learn’s case that make this difficult.

Labels: New Feature, Needs Decision - Include Feature
Enable exporting trained models to text files to be able to import later

### Describe the workflow you want to enable

```python
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor

# make data
X,y = fetch_california_housing(return_X_y=True)

# instantiate Random-Forest and fit it
rf_model = RandomForestRegressor(min_samples_leaf=5, random_state=0, n_jobs=-1)
rf_model.fit(X, y)

# export model to a text file, inspired by https://xgboost.readthedocs.io/en/release_3.0.0/python/python_api.html#xgboost.XGBRegressor.save_model
rf_model.save_model("model.json")


#################### in a new python environment
from sklearn.ensemble import RandomForestRegressor
rf_model2 = RandomForestRegressor(min_samples_leaf=5, random_state=0, n_jobs=-1)
rf_model2.load_model("model.json")
```

### Describe your proposed solution

The current recommended way I believe is to export fit (or trained) models is to serialize them using joblib, which depends on python version, joblib version and scikit-learn version too, and I presume this may lead to issues with OS and CPU architecture as well (windows or  GNU Linux and x86 or ARM64).

So, I request a way to export model's trained weights (or other relevant things like bins & trees for RandomForestRegressor) for it be to loaded from any scikit-learn version or python version or operating system. This is just how the big packages like [xgboost](https://xgboost.readthedocs.io/en/release_3.0.0/python/python_api.html#xgboost.XGBRegressor.save_model) and pytorch ([using `state_dict`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.load_state_dict)) and hence transformers,  handle things. 

This would enable to change environments and platforms easily without having to train model for the new package-versions, architecture and OS again or every time an update in them is required.

### Describe alternatives you've considered, if relevant

There is no alternative to everything that scikit-learn offers as of now.

### Additional context

This has been provied above.

COMMENT:
The main source of extra maintenance is setting up and running the infrastructure that checks that internal changes in scikit-learn code do not break model persisted with older versions.

Most of the time there will be no problem. Most pickle or skops.io persisted models load from one version to the next most of the time. What is costly is setting up and maintaining the test infrastructure to check whether this is actually the case (and updating the migration tool accordingly when needed, before a new scikit-learn release comes out). As long a we do not have this infrastructure in place, we cannot guarantee that persisted model will load from one scikit-learn version to the next.

Labels: New Feature, Needs Decision - Include Feature
Enable exporting trained models to text files to be able to import later

### Describe the workflow you want to enable

```python
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor

# make data
X,y = fetch_california_housing(return_X_y=True)

# instantiate Random-Forest and fit it
rf_model = RandomForestRegressor(min_samples_leaf=5, random_state=0, n_jobs=-1)
rf_model.fit(X, y)

# export model to a text file, inspired by https://xgboost.readthedocs.io/en/release_3.0.0/python/python_api.html#xgboost.XGBRegressor.save_model
rf_model.save_model("model.json")


#################### in a new python environment
from sklearn.ensemble import RandomForestRegressor
rf_model2 = RandomForestRegressor(min_samples_leaf=5, random_state=0, n_jobs=-1)
rf_model2.load_model("model.json")
```

### Describe your proposed solution

The current recommended way I believe is to export fit (or trained) models is to serialize them using joblib, which depends on python version, joblib version and scikit-learn version too, and I presume this may lead to issues with OS and CPU architecture as well (windows or  GNU Linux and x86 or ARM64).

So, I request a way to export model's trained weights (or other relevant things like bins & trees for RandomForestRegressor) for it be to loaded from any scikit-learn version or python version or operating system. This is just how the big packages like [xgboost](https://xgboost.readthedocs.io/en/release_3.0.0/python/python_api.html#xgboost.XGBRegressor.save_model) and pytorch ([using `state_dict`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.load_state_dict)) and hence transformers,  handle things. 

This would enable to change environments and platforms easily without having to train model for the new package-versions, architecture and OS again or every time an update in them is required.

### Describe alternatives you've considered, if relevant

There is no alternative to everything that scikit-learn offers as of now.

### Additional context

This has been provied above.

COMMENT:
> In the company I work for, we have a few ML models trained using Scikit-learn that are currently in production. These models rarely require updates, so retraining is largely out of scope. However, over time, due to security and maintenance requirements, we need to upgrade the production environment — including the Python and Scikit-learn versions. Ideally, we’d like to upgrade the environment without needing to retrain the models.

This is perfectly valid need and others have expressed similar needs in the past. If your company is willing to sponsor the development and maintenance of that testing infrastruction and migration tool, please reach to us via the link above.

Labels: New Feature, Needs Decision - Include Feature