Repository: scikit-learn/scikit-learn
Pull request: ENH Fix AttributeError of compound kernel of Gaussian processes
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

Fixes #21973


#### What does this implement/fix? Explain your changes.

Raise better error if `CompoundKernel` is passed to `GaussianProcessClassifier` as kernel

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->
This was originally proposed in [scikit-learn/issues/21973#issuecomment-996936548](https://github.com/scikit-learn/scikit-learn/issues/21973#issuecomment-996936548)

Problem statement:
AttributeError of compound kernel of Gaussian processes 
### Describe the bug

I am trying to use GaussianProcess classifier or regressor using the compound kernel (comprised of RBF and white kernels). Fit method of Gaussian process generates an error regarding its kernel, declaring that 'CompoundKernel' object has no attribute 'k1'. I regenerated the error using the following simpler code:



### Steps/Code to Reproduce

```python
import numpy as np
import pandas as pd
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import CompoundKernel, WhiteKernel, RBF
X1 = np.random.exponential(range(0,10), size = 10)
X2 = np.random.poisson(range(0,10), size = 10)
y = np.random.normal(size = 10)
df = pd.DataFrame()
df["X1"] = X1
df["X2"] = X2
df["y"] = y
df1 = df.iloc[:,0:2]
df2 = df.iloc[:,2]

kernel = CompoundKernel([WhiteKernel(noise_level=2), RBF(length_scale=3)])
gaus = GaussianProcessRegressor(kernel = kernel)
gaus.fit(df1, df2) # executing this line generates the error
```

### Expected Results

```pytb
AttributeError                            Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_2056/1607188154.py in <module>
     15 kernel = CompoundKernel([WhiteKernel(noise_level=2), RBF(length_scale=3)])
     16 gaus = GaussianProcessRegressor(kernel = kernel)
---> 17 gaus.fit(df1, df2) # executing this line generates the error
......
......
    540             The non-fixed, log-transformed hyperparameters of the kernel
    541         """
--> 542         k_dims = self.k1.n_dims
    543         for i, kernel in enumerate(self.kernels):
    544             kernel.theta = theta[i * k_dims:(i + 1) * k_dims]

AttributeError: 'CompoundKernel' object has no attribute 'k1'
```

### Actual Results

The code works well without using the compound kernel (e.g., RBF only), and the way I built the compound kernel is exactly according to the provided format of sklearn. Not sure why I receive this attribute error. 

### Versions

```
System:
    python: 3.7.12 (default, Sep 10 2021, 00:21:48)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.1
        numpy: 1.19.5
        scipy: 1.4.1
       Cython: 0.29.24
       pandas: 1.1.5
   matplotlib: 3.2.2
       joblib: 1.1.0
threadpoolctl: 3.0.0

Built with OpenMP: True
```

Hints from issue discussion:
I think that this is expected. `CompoundKernel` is used internally for classification: we will train a kernel per class in the multiclass setting and we store them in a `CompoundKernel`.

In regression, you don't have such behaviour: you have a single kernel and a single estimator.
I think what you try to do is to combine the kernel and for this purpose, you only need to use the arithmetic operator.
You can check the example of the CO2 data:

https://scikit-learn.org/dev/auto_examples/gaussian_process/plot_gpr_co2.html#sphx-glr-auto-examples-gaussian-process-plot-gpr-co2-py

where we combine several kernels together to get the different trends of the time series.
Thank you for your clarification. Using the arithmetic operator as suggested in the sklearn documentation solves the issue, but using this `CompoundKernel` module seems to generate error for a classification problem in the same manner. 
> this CompoundKernel module seems to generate error for a classification problem in the same manner

I was probably not clear, you should not use it even in classification. Basically, it is only used in the internal of scikit-learn:

```python
In [1]: from sklearn.datasets import load_iris
   ...: from sklearn.gaussian_process import GaussianProcessClassifier
   ...: from sklearn.gaussian_process.kernels import RBF
   ...: X, y = load_iris(return_X_y=True)
   ...: kernel = 1.0 * RBF(1.0)
   ...: gpc = GaussianProcessClassifier(kernel=kernel,
   ...:         random_state=0).fit(X, y)
In [2]: gpc.kernel_
Out[2]: CompoundKernel(7.71, 1.36, 5.26, 0.669, 6.13, 1.15)
In [3]: gpc.kernel_.get_params()
Out[3]: 
{'kernels': [47.2**2 * RBF(length_scale=3.92),
  13.8**2 * RBF(length_scale=1.95),
  21.4**2 * RBF(length_scale=3.15)]}
```

Here we indeed have 3 kernels, because the model is trained in a 1-vs-rest manner, thus 1 estimator + 1 kernel for each. But it is transparent to the user.
I got the point now. Thank you. 

> > this CompoundKernel module seems to generate error for a classification problem in the same manner
> 
> I was probably not clear, you should not use it even in classification. Basically, it is only used in the internal of scikit-learn:
> 
> ```python
> In [1]: from sklearn.datasets import load_iris
>    ...: from sklearn.gaussian_process import GaussianProcessClassifier
>    ...: from sklearn.gaussian_process.kernels import RBF
>    ...: X, y = load_iris(return_X_y=True)
>    ...: kernel = 1.0 * RBF(1.0)
>    ...: gpc = GaussianProcessClassifier(kernel=kernel,
>    ...:         random_state=0).fit(X, y)
> In [2]: gpc.kernel_
> Out[2]: CompoundKernel(7.71, 1.36, 5.26, 0.669, 6.13, 1.15)
> In [3]: gpc.kernel_.get_params()
> Out[3]: 
> {'kernels': [47.2**2 * RBF(length_scale=3.92),
>   13.8**2 * RBF(length_scale=1.95),
>   21.4**2 * RBF(length_scale=3.15)]}
> ```
> 
> Here we indeed have 3 kernels, because the model is trained in a 1-vs-rest manner, thus 1 estimator + 1 kernel for each. But it is transparent to the user.


@glemaitre Should we raise a better error when `CompoundKernel` is passed into `GaussianProcessClassifier`?
Yes indeed, we could make some improvement in the error message to catch it early in `fit`.
We should as well improve our documentation to mention not to use it and state where this is used.
@glemaitre Can i work on this?

take

Commit message: removed spacebar in docs

Patch to review:
diff --git a/doc/whats_new/v1.1.rst b/doc/whats_new/v1.1.rst
index eef1135855560..4e26134bec38a 100644
--- a/doc/whats_new/v1.1.rst
+++ b/doc/whats_new/v1.1.rst
@@ -273,6 +273,13 @@ Changelog
   F-statistic).
   :pr:`17819` by :user:`Juan Carlos Alfaro Jiménez <alfaro96>`.
 
+:mod:`sklearn.gaussian_process`
+...............................
+
+- |Fix| Raise better error if `CompoundKernel is passed to 
+  `GaussianProcessClassifier` as kernel .
+  :pr:`22223` by :user:`MarcoM <marcozzxx810>`.
+
 :mod:`sklearn.impute`
 .....................
 
diff --git a/sklearn/gaussian_process/_gpc.py b/sklearn/gaussian_process/_gpc.py
index cdd79b9fde26e..8052d229f4027 100644
--- a/sklearn/gaussian_process/_gpc.py
+++ b/sklearn/gaussian_process/_gpc.py
@@ -503,7 +503,8 @@ class GaussianProcessClassifier(ClassifierMixin, BaseEstimator):
     kernel : kernel instance, default=None
         The kernel specifying the covariance function of the GP. If None is
         passed, the kernel "1.0 * RBF(1.0)" is used as default. Note that
-        the kernel's hyperparameters are optimized during fitting.
+        the kernel's hyperparameters are optimized during fitting. Also kernel
+        cannot be a CompoundKernel as it is a private API.
 
     optimizer : 'fmin_l_bfgs_b' or callable, default='fmin_l_bfgs_b'
         Can either be one of the internally supported optimizers for optimizing
@@ -673,6 +674,9 @@ def fit(self, X, y):
         self : object
             Returns an instance of self.
         """
+        if isinstance(self.kernel, CompoundKernel):
+            raise ValueError("kernel cannot be a CompoundKernel")
+
         if self.kernel is None or self.kernel.requires_vector_input:
             X, y = self._validate_data(
                 X, y, multi_output=False, ensure_2d=True, dtype="numeric"
diff --git a/sklearn/gaussian_process/tests/test_gpc.py b/sklearn/gaussian_process/tests/test_gpc.py
index be7d365601f87..c45ba35ae37d7 100644
--- a/sklearn/gaussian_process/tests/test_gpc.py
+++ b/sklearn/gaussian_process/tests/test_gpc.py
@@ -11,7 +11,12 @@
 import pytest
 
 from sklearn.gaussian_process import GaussianProcessClassifier
-from sklearn.gaussian_process.kernels import RBF, ConstantKernel as C, WhiteKernel
+from sklearn.gaussian_process.kernels import (
+    RBF,
+    CompoundKernel,
+    ConstantKernel as C,
+    WhiteKernel,
+)
 from sklearn.gaussian_process.tests._mini_sequence_kernel import MiniSeqKernel
 from sklearn.exceptions import ConvergenceWarning
 
@@ -260,3 +265,20 @@ def test_warning_bounds():
         "Increasing the bound and calling "
         "fit again may find a better value."
     )
+
+
+@pytest.mark.parametrize(
+    "params, TypeError, err_msg",
+    [
+        (
+            {"kernel": CompoundKernel(0)},
+            ValueError,
+            "kernel cannot be a CompoundKernel",
+        )
+    ],
+)
+def test_gpc_fit_error(params, TypeError, err_msg):
+    """Check that expected error are raised during fit."""
+    gpc = GaussianProcessClassifier(**params)
+    with pytest.raises(TypeError, match=err_msg):
+        gpc.fit(X, y)


Code review comments:
diff_hunk: @@ -260,3 +265,20 @@ def test_warning_bounds():
         "Increasing the bound and calling "
         "fit again may find a better value."
     )
+
+
+@pytest.mark.parametrize(
+    "params, TypeError, err_msg",
path: sklearn/gaussian_process/tests/test_gpc.py
text: @user1:
I see this is a copy of:

https://github.com/scikit-learn/scikit-learn/blob/1d1aadd0711b87d2a11c80aad15df6f8cf156712/sklearn/gaussian_process/tests/test_gpr.py#L632

but I prefer to use something like this:

```suggestion
    "params, error_type, err_msg",
```

since `TypeError` is a already a name for a Python exception.

@author:
do I need to edit the `test_gpr.py` too?

diff_hunk: @@ -503,7 +503,8 @@ class GaussianProcessClassifier(ClassifierMixin, BaseEstimator):
     kernel : kernel instance, default=None
         The kernel specifying the covariance function of the GP. If None is
         passed, the kernel "1.0 * RBF(1.0)" is used as default. Note that
-        the kernel's hyperparameters are optimized during fitting.
+        the kernel's hyperparameters are optimized during fitting. Also kernel
+        cannot be a CompoundKernel as it is a private API.
path: sklearn/gaussian_process/_gpc.py
text: @user1:
Strictly speaking CompoundKernel is public API since it has it's own page in the [public API docs](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.kernels.CompoundKernel.html#sklearn.gaussian_process.kernels.CompoundKernel)

```suggestion
        cannot be a `CompoundKernel`.
```

**Edit** I mistype in my previous message. I meant to say that `CompoundKernel` is public.

@author:
sure, I will follow your suggestion

diff_hunk: @@ -260,3 +265,20 @@ def test_warning_bounds():
         "Increasing the bound and calling "
         "fit again may find a better value."
     )
+
+
+@pytest.mark.parametrize(
+    "params, TypeError, err_msg",
+    [
+        (
+            {"kernel": CompoundKernel(0)},
+            ValueError,
+            "kernel cannot be a CompoundKernel",
+        )
+    ],
+)
+def test_gpc_fit_error(params, TypeError, err_msg):
+    """Check that expected error are raised during fit."""
+    gpc = GaussianProcessClassifier(**params)
+    with pytest.raises(TypeError, match=err_msg):
path: sklearn/gaussian_process/tests/test_gpc.py
text: @user1:
```suggestion
def test_gpc_fit_error(params, error_type, err_msg):
    """Check that expected error are raised during fit."""
    gpc = GaussianProcessClassifier(**params)
    with pytest.raises(error_type, match=err_msg):
```