Issue 1:
Describe the bug

When fitting a GPR with y-values having std=0 (and normalize=True), an error occurs. The error relates to dividing by 0 in sklearn/gaussian_processes/_gpr.py:

scikit-learn/sklearn/gaussian_process/_gpr.py

Line 201
in
fd1ff73

y = (y - self._y_train_mean) / self._y_train_std

Steps/Code to Reproduce

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import ConstantKernel, Matern

X = [[1.,2.],[2.,2.]]
y = [0,0]

m52 = ConstantKernel(1.0) * Matern(nu=2.5)
gpr = GaussianProcessRegressor(kernel=m52, normalize_y=True, n_restarts_optimizer=3)

gpr.fit(X,y)

Expected Results

Either the regressor could internally switch normalize_y to False, when np.std(y)==0 or set the std to 1. Or warn the user of the eminent error instead of an error message that can be difficult to understand.

Actual Results

"runfile('C:/users/sqbl/onedrive - novo nordisk/dokumenter/python/12 gitlab inout/po20200908/processoptimizer/ProcessOptimizer/learning/gaussian_process/untitled0.py', wdir='C:/users/sqbl/onedrive - novo nordisk/dokumenter/python/12 gitlab inout/po20200908/processoptimizer/ProcessOptimizer/learning/gaussian_process', current_namespace=True)

C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\sklearn\gaussian_process_gpr.py:201: RuntimeWarning: invalid value encountered in true_divide

y = (y - self._y_train_mean) / self._y_train_std

Traceback (most recent call last):

File "C:\users\sqbl\onedrive - novo nordisk\dokumenter\python\12 gitlab inout\po20200908\processoptimizer\ProcessOptimizer\learning\gaussian_process\untitled0.py", line 16, in

gpr.fit(X,y)

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\sklearn\gaussian_process_gpr.py", line 234, in fit

self.kernel_.bounds))]

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\sklearn\gaussian_process_gpr.py", line 503, in _constrained_optimization

bounds=bounds)

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\scipy\optimize_minimize.py", line 618, in minimize

callback=callback, **options)

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\scipy\optimize\lbfgsb.py", line 308, in _minimize_lbfgsb

finite_diff_rel_step=finite_diff_rel_step)

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\scipy\optimize\optimize.py", line 262, in _prepare_scalar_function

finite_diff_rel_step, bounds, epsilon=epsilon)

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\scipy\optimize_differentiable_functions.py", line 76, in init

self._update_fun()

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\scipy\optimize_differentiable_functions.py", line 166, in _update_fun

self._update_fun_impl()

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\scipy\optimize_differentiable_functions.py", line 73, in update_fun

self.f = fun_wrapped(self.x)

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\scipy\optimize_differentiable_functions.py", line 70, in fun_wrapped

return fun(x, *args)

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\scipy\optimize\optimize.py", line 74, in call

self._compute_if_needed(x, *args)

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\scipy\optimize\optimize.py", line 68, in _compute_if_needed

fg = self.fun(x, *args)

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\sklearn\gaussian_process_gpr.py", line 225, in obj_func

theta, eval_gradient=True, clone_kernel=False)

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\sklearn\gaussian_process_gpr.py", line 476, in log_marginal_likelihood

alpha = cho_solve((L, True), y_train) # Line 3

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\scipy\linalg\decomp_cholesky.py", line 194, in cho_solve

b1 = asarray_chkfinite(b)

File "C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\lib\site-packages\numpy\lib\function_base.py", line 486, in asarray_chkfinite

"array must not contain infs or NaNs")

ValueError: array must not contain infs or NaNs"

Versions

System:

python: 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)]

executable: C:\Users\sqbl\AppData\Local\Continuum\anaconda3\envs\updatePO20200908\python.exe

machine: Windows-10-10.0.17763-SP0

Python dependencies:

pip: 20.2.2

setuptools: 41.6.0

sklearn: 0.23.2

numpy: 1.19.1

scipy: 1.5.2

Cython: None

pandas: 1.1.2

matplotlib: 3.3.1

joblib: 0.16.0

threadpoolctl: 2.1.0

Built with OpenMP: True

Issue 2:
In #15782 (comment), @rkern writes regarding that fix:

FWIW, this broke a few downstream users where y_train.std() == 0 (e.g. only one datapoint) when being used for Bayesian optimization. This probably needs a guard for such a case (which is common in the Bayesian optimization use case)

More detail in bayesian-optimization/BayesianOptimization#243 (comment):

The normalize_y=True option which is used now divides out the standard deviation of the y data, not just subtracting the mean. When there is just one data point, this results in a NaN.