Repository: cornellius-gp/gpytorch

Issue:
[Bug] Wrong formulation of the PeriodicKernel
# 🐛 Bug

The current formulation of the periodic kernel is the following:

<img src="https://render.githubusercontent.com/render/math?math=k(\mathbf{x^1, x^2}) = \exp ( \frac{-2 \sin^2(\frac{\pi}{p} \sum_i | x^{1}_{i} - x^{2}_{i} |)}{l^2} )">

However, the original formula proposed [Mackay, 1998](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.81.1927&rep=rep1&type=pdf) (Eq. 47) is the following:

<img src="https://render.githubusercontent.com/render/math?math=k(\mathbf{x^1, x^2}) = \exp ( \frac{-0.5  \sum_i \sin^2(\frac{\pi}{p} | x^{1}_{i} - x^{2}_{i} |)}{l^2} )">

The constant is not -2 or -0.5 is not really important (it will have an impact on the derivative only). However, the original formulation is taking the the sum of the sine squared of the difference. Currently, the formulation is the sine squared of the sum of the difference. Note that the confusion could come from the fact that in the literature, the kernel is used for 1D signal and thus there is no sum, thus both formulations are equivalent.

In addition, looking at the implementation, it seems that the distance computed between the samples is norm-2 distance and not the norm-1 distance.

The current formulation make it that the kernel is not PSD. In addition, there is still another bug with a missing square over the length scale as reported in https://github.com/cornellius-gp/gpytorch/issues/1020.

Note: I found this issue because I am currently working on a similar issues in scikit-learn:
https://github.com/scikit-learn/scikit-learn/issues/19343
https://github.com/scikit-learn/scikit-learn/pull/20070

## To reproduce

** Code snippet to reproduce **
```python
import numpy as np
import torch
from gpytorch.kernels import PeriodicKernel

r = np.linspace(0, 1, num=4)
train_x, train_y = np.meshgrid(r, r)
X = np.stack((train_x.flatten(), train_y.flatten()), axis=-1)

periodicity, length_scale=1.5, 0.5
kernel_pytorch = PeriodicKernel(
    period=periodicity, length_scale=length_scale
)
K_pytorch = kernel_pytorch(torch.from_numpy(X))

np.linalg.eigh(K_pytorch.numpy())[0]
```

** Stack trace/error message **
```
array([-1.27815495, -1.21435487, -1.21435487, -0.93566031, -0.75456704,
       -0.24301193, -0.24301193,  0.47435512,  0.72370254,  1.58773803,
        1.58773803,  2.42356711,  2.67356606,  2.67356606,  3.10565507,
        6.63322787])
```

There is some negative eigenvalues.

## Expected Behavior

We are investigating the issue in scikit-learn and already implemented a naive approach: https://github.com/scikit-learn/scikit-learn/pull/20070 for tracking the bug. With the code of the PR, there is no negative eigenvalues:

```python
from sklearn.gaussian_process.kernels import ExpSineSquared

kernel_sklearn = ExpSineSquared(
    periodicity=periodicity, length_scale=length_scale,
)
K_sklearn = kernel_sklearn(X)
np.linalg.eigh(K_sklearn)[0]
```

```
array([0.88422032, 0.92031869, 0.92031869, 0.95789078, 0.95961332,
       0.95961332, 0.99716727, 0.99716727, 0.99878962, 0.99878962,
       1.03787671, 1.03787671, 1.0414347 , 1.08219069, 1.08219069,
       1.12454163])
```

On the process, we also worked with the implementation of tensorflow-probability that follow the equation of the paper, and we have consistent results:

```python
from tensorflow_probability import math

kernel_tfp = math.psd_kernels.ExpSinSquared(
   period=periodicity, length_scale=length_scale
)
K_tfp = kernel_tfp.matrix(X, X)
np.linalg.eigh(K_tfp.numpy())[0]
``` 

```
array([0.8842203 , 0.92031866, 0.92031866, 0.95789075, 0.9596133 ,
       0.9596133 , 0.9971673 , 0.9971673 , 0.9987896 , 0.9987896 ,
       1.0378767 , 1.0378767 , 1.0414346 , 1.0821906 , 1.0821906 ,
       1.1245416 ], dtype=float32)
```

## System information

**Please complete the following information:**
- GPyTorch version: `1.4.1`
- PyTorch version: `1.8.1+cpu`
- Linux 5.8.0-50-generic #56-Ubuntu SMP Mon Apr 12 17:18:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux



Discussion:

Resolving patch:
diff --git a/gpytorch/kernels/periodic_kernel.py b/gpytorch/kernels/periodic_kernel.py
--- a/gpytorch/kernels/periodic_kernel.py
+++ b/gpytorch/kernels/periodic_kernel.py
@@ -14,16 +14,22 @@ class PeriodicKernel(Kernel):
 
     .. math::
 
-       \begin{equation*}
-          k_{\text{Periodic}}(\mathbf{x_1}, \mathbf{x_2}) = \exp \left(
-            \frac{2 \sin^2 \left( \pi \Vert \mathbf{x_1} - \mathbf{x_2} \Vert_1 / p \right) }
-            { \ell^2 } \right)
-       \end{equation*}
+        \begin{equation*}
+            k_{\text{Periodic}}(\mathbf{x_1}, \mathbf{x_2}) = \exp \left(
+            -2 \sum_i
+            \frac{\sin ^2 \left( \frac{\pi}{p} (\mathbf{x_{1,i}} - \mathbf{x_{2,i}} ) \right)}{\lambda}
+            \right)
+        \end{equation*}
 
     where
 
-    * :math:`p` is the periord length parameter.
-    * :math:`\ell` is a lengthscale parameter.
+    * :math:`p` is the period length parameter.
+    * :math:`\lambda` is a lengthscale parameter.
+
+    Equation is based on [David Mackay's Introduction to Gaussian Processes equation 47]
+    (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.81.1927&rep=rep1&type=pdf)
+    albeit without feature-specific lengthscales and period lengths. The exponential
+    coefficient was changed and lengthscale is not squared to maintain backwards compatibility
 
     .. note::
 
@@ -32,7 +38,7 @@ class PeriodicKernel(Kernel):
 
     .. note::
 
-        This kernel does not have an ARD lengthscale option.
+        This kernel does not have an ARD lengthscale or period length option.
 
     Args:
         :attr:`batch_shape` (torch.Size, optional):
@@ -107,10 +113,10 @@ def _set_period_length(self, value):
         self.initialize(raw_period_length=self.raw_period_length_constraint.inverse_transform(value))
 
     def forward(self, x1, x2, diag=False, **params):
-        x1_ = x1.div(self.period_length)
-        x2_ = x2.div(self.period_length)
-        diff = self.covar_dist(x1_, x2_, diag=diag, **params)
-        res = torch.sin(diff.mul(math.pi)).pow(2).mul(-2 / self.lengthscale).exp_()
+        x1_ = x1.div(self.period_length).mul(math.pi)
+        x2_ = x2.div(self.period_length).mul(math.pi)
+        diff = x1_.unsqueeze(-2) - x2_.unsqueeze(-3)
+        res = diff.sin().pow(2).sum(dim=-1).div(self.lengthscale).mul(-2.0).exp_()
         if diag:
             res = res.squeeze(0)
         return res


Test patch:
diff --git a/test/kernels/test_periodic_kernel.py b/test/kernels/test_periodic_kernel.py
--- a/test/kernels/test_periodic_kernel.py
+++ b/test/kernels/test_periodic_kernel.py
@@ -26,6 +26,25 @@ def test_computes_periodic_function(self):
         res = kernel(a, b).evaluate()
         self.assertLess(torch.norm(res - actual), 1e-5)
 
+    def test_is_pd(self):
+        # ensures 1d input is positive definite with additional jitter
+        x = torch.randn(100).reshape(-1, 1)
+        kernel = PeriodicKernel()
+        with torch.no_grad():
+            K = kernel(x, x).evaluate() + 1e-4 * torch.eye(len(x))
+            eig = torch.symeig(K)[0]
+            self.assertTrue((eig > 0.0).all().item())
+
+    def test_multidimensional_inputs(self):
+        # test taken from issue #835
+        # ensures multidimensional input results in a positive definite kernel matrix, with additional jitter
+        x = torch.randn(1000, 2)
+        kernel = PeriodicKernel()
+        with torch.no_grad():
+            K = kernel(x, x).evaluate() + 1e-4 * torch.eye(len(x))
+            eig = torch.symeig(K)[0]
+            self.assertTrue((eig > 0.0).all().item())
+
     def test_batch(self):
         a = torch.tensor([[4, 2, 8], [1, 2, 3]], dtype=torch.float).view(2, 3, 1)
         b = torch.tensor([[0, 2], [-1, 2]], dtype=torch.float).view(2, 2, 1)
@@ -36,10 +55,7 @@ def test_batch(self):
 
         actual = torch.zeros(2, 3, 2)
         for k in range(2):
-            for i in range(3):
-                for j in range(2):
-                    val = 2 * torch.pow(torch.sin(math.pi * (a[k, i] - b[k, j]) / period), 2) / lengthscale
-                    actual[k, i, j] = torch.exp(-val).item()
+            actual[k] = kernel(a[k], b[k]).evaluate()
 
         res = kernel(a, b).evaluate()
         self.assertLess(torch.norm(res - actual), 1e-5)
@@ -54,10 +70,9 @@ def test_batch_separate(self):
 
         actual = torch.zeros(2, 3, 2)
         for k in range(2):
-            for i in range(3):
-                for j in range(2):
-                    val = 2 * torch.pow(torch.sin(math.pi * (a[k, i] - b[k, j]) / period[k]), 2) / lengthscale[k]
-                    actual[k, i, j] = torch.exp(-val).item()
+            diff = a[k].unsqueeze(1) - b[k].unsqueeze(0)
+            diff = diff * math.pi / period[k].unsqueeze(-1)
+            actual[k] = diff.sin().pow(2).sum(dim=-1).div(lengthscale[k]).mul(-2.0).exp_()
 
         res = kernel(a, b).evaluate()
         self.assertLess(torch.norm(res - actual), 1e-5)