Repository: cornellius-gp/gpytorch

Issue:
[Bug] gpytorch PeriodicKernel, diag() method yields to RuntimeError message
# 🐛 Bug

Hello, 
I have a problem with the gpytorch Periodickernel.
When triying to call the diag() method, it yiels to a runtime error.
This leads to the problem that no confidence region or predictive variance can be computed later for predictions.
That happens both for few and large data.
I am not sure if this is a bug or am I doing something wrong here?

## To reproduce

** Code snippet to reproduce **
covar_module = gpytorch.kernels.PeriodicKernel()
x1 = torch.randn(50)
lazy_covar_matrix = covar_module(x1) # Returns a RootLazyTensor
lazy_covar_matrix.diag()

** Stack trace/error message **
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_43664/910117771.py in <module>
      3 x1 = torch.randn(50)
      4 lazy_covar_matrix = covar_module(x1) # Returns a RootLazyTensor
----> 5 lazy_covar_matrix.diag()

~/anaconda3/lib/python3.8/site-packages/gpytorch/utils/memoize.py in g(self, *args, **kwargs)
     57         kwargs_pkl = pickle.dumps(kwargs)
     58         if not _is_in_cache(self, cache_name, *args, kwargs_pkl=kwargs_pkl):
---> 59             return _add_to_cache(self, cache_name, method(self, *args, **kwargs), *args, kwargs_pkl=kwargs_pkl)
     60         return _get_from_cache(self, cache_name, *args, kwargs_pkl=kwargs_pkl)
     61 

~/anaconda3/lib/python3.8/site-packages/gpytorch/lazy/lazy_evaluated_kernel_tensor.py in diag(self)
    309             expected_shape = self.shape[:-1]
    310             if res.shape != expected_shape:
--> 311                 raise RuntimeError(
    312                     "The kernel {} is not equipped to handle and diag. Expected size {}. "
    313                     "Got size {}".format(self.kernel.__class__.__name__, expected_shape, res.shape)

RuntimeError: The kernel PeriodicKernel is not equipped to handle and diag. Expected size torch.Size([50]). Got size torch.Size([50, 50])

## Expected Behavior

Should return the diagonal of the kernel.

## System information

**Please complete the following information:**
GPyTorch Version (run `print(gpytorch.__version__)` 
1.6.0
PyTorch Version (run `print(torch.__version__)` 
1.10.0
Computer OS
Windows 10



Discussion:
Thanks for catching this. I'm taking a look at this.

Resolving patch:
diff --git a/gpytorch/kernels/periodic_kernel.py b/gpytorch/kernels/periodic_kernel.py
--- a/gpytorch/kernels/periodic_kernel.py
+++ b/gpytorch/kernels/periodic_kernel.py
@@ -17,9 +17,9 @@ class PeriodicKernel(Kernel):
     .. math::
 
         \begin{equation*}
-            k_{\text{Periodic}}(\mathbf{x_1}, \mathbf{x_2}) = \exp \left(
+            k_{\text{Periodic}}(\mathbf{x}, \mathbf{x'}) = \exp \left(
             -2 \sum_i
-            \frac{\sin ^2 \left( \frac{\pi}{p} (\mathbf{x_{1,i}} - \mathbf{x_{2,i}} ) \right)}{\lambda}
+            \frac{\sin ^2 \left( \frac{\pi}{p} ({x_{i}} - {x_{i}'} ) \right)}{\lambda}
             \right)
         \end{equation*}
 
@@ -28,9 +28,8 @@ class PeriodicKernel(Kernel):
     * :math:`p` is the period length parameter.
     * :math:`\lambda` is a lengthscale parameter.
 
-    Equation is based on [David Mackay's Introduction to Gaussian Processes equation 47]
-    (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.81.1927&rep=rep1&type=pdf)
-    albeit without feature-specific lengthscales and period lengths. The exponential
+    Equation is based on `David Mackay's Introduction to Gaussian Processes equation 47`_
+    (albeit without feature-specific lengthscales and period lengths). The exponential
     coefficient was changed and lengthscale is not squared to maintain backwards compatibility
 
     .. note::
@@ -38,34 +37,35 @@ class PeriodicKernel(Kernel):
         This kernel does not have an `outputscale` parameter. To add a scaling parameter,
         decorate this kernel with a :class:`gpytorch.kernels.ScaleKernel`.
 
-    .. note::
-
-        This kernel does not have an ARD lengthscale or period length option.
-
-    Args:
-        :attr:`batch_shape` (torch.Size, optional):
-            Set this if you want a separate lengthscale for each
-             batch of input data. It should be `b` if :attr:`x1` is a `b x n x d` tensor. Default: `torch.Size([])`.
-        :attr:`active_dims` (tuple of ints, optional):
-            Set this if you want to compute the covariance of only a few input dimensions. The ints
-            corresponds to the indices of the dimensions. Default: `None`.
-        :attr:`period_length_prior` (Prior, optional):
-            Set this if you want to apply a prior to the period length parameter.  Default: `None`.
-        :attr:`lengthscale_prior` (Prior, optional):
-            Set this if you want to apply a prior to the lengthscale parameter.  Default: `None`.
-        :attr:`lengthscale_constraint` (Constraint, optional):
-            Set this if you want to apply a constraint to the value of the lengthscale. Default: `Positive`.
-        :attr:`period_length_constraint` (Constraint, optional):
-            Set this if you want to apply a constraint to the value of the period length. Default: `Positive`.
-        :attr:`eps` (float):
-            The minimum value that the lengthscale/period length can take
-            (prevents divide by zero errors). Default: `1e-6`.
-
-    Attributes:
-        :attr:`lengthscale` (Tensor):
-            The lengthscale parameter. Size = `*batch_shape x 1 x 1`.
-        :attr:`period_length` (Tensor):
-            The period length parameter. Size = `*batch_shape x 1 x 1`.
+    :param ard_num_dims: (Default: `None`) Set this if you want a separate lengthscale for each
+        input dimension. It should be `d` if :attr:`x1` is a `... x n x d` matrix.
+    :type ard_num_dims: int, optional
+    :param batch_shape: (Default: `None`) Set this if you want a separate lengthscale for each
+         batch of input data. It should be `torch.Size([b1, b2])` for a `b1 x b2 x n x m` kernel output.
+    :type batch_shape: torch.Size, optional
+    :param active_dims: (Default: `None`) Set this if you want to
+        compute the covariance of only a few input dimensions. The ints
+        corresponds to the indices of the dimensions.
+    :type active_dims: Tuple(int)
+    :param period_length_prior: (Default: `None`)
+        Set this if you want to apply a prior to the period length parameter.
+    :type period_length_prior: ~gpytorch.priors.Prior, optional
+    :param period_length_constraint: (Default: `Positive`) Set this if you want
+        to apply a constraint to the period length parameter.
+    :type period_length_constraint: ~gpytorch.constraints.Interval, optional
+    :param lengthscale_prior: (Default: `None`)
+        Set this if you want to apply a prior to the lengthscale parameter.
+    :type lengthscale_prior: ~gpytorch.priors.Prior, optional
+    :param lengthscale_constraint: (Default: `Positive`) Set this if you want
+        to apply a constraint to the lengthscale parameter.
+    :type lengthscale_constraint: ~gpytorch.constraints.Interval, optional
+    :param eps: (Default: 1e-6) The minimum value that the lengthscale can take (prevents divide by zero errors).
+    :type eps: float, optional
+
+    :var torch.Tensor lengthscale: The lengthscale parameter. Size/shape of parameter depends on the
+        :attr:`ard_num_dims` and :attr:`batch_shape` arguments.
+    :var torch.Tensor period_length: The period length parameter. Size/shape of parameter depends on the
+        :attr:`ard_num_dims` and :attr:`batch_shape` arguments.
 
     Example:
         >>> x = torch.randn(10, 5)
@@ -78,6 +78,9 @@ class PeriodicKernel(Kernel):
         >>> # Batch: different lengthscale for each batch
         >>> covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.PeriodicKernel(batch_size=2))
         >>> covar = covar_module(x)  # Output: LazyVariable of size (2 x 10 x 10)
+
+    .. _David Mackay's Introduction to Gaussian Processes equation 47:
+        http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.81.1927&rep=rep1&type=pdf
     """
 
     has_lengthscale = True
@@ -92,8 +95,9 @@ def __init__(
         if period_length_constraint is None:
             period_length_constraint = Positive()
 
+        ard_num_dims = kwargs.get("ard_num_dims", 1)
         self.register_parameter(
-            name="raw_period_length", parameter=torch.nn.Parameter(torch.zeros(*self.batch_shape, 1, 1))
+            name="raw_period_length", parameter=torch.nn.Parameter(torch.zeros(*self.batch_shape, 1, ard_num_dims))
         )
 
         if period_length_prior is not None:
@@ -122,10 +126,23 @@ def _set_period_length(self, value):
         self.initialize(raw_period_length=self.raw_period_length_constraint.inverse_transform(value))
 
     def forward(self, x1, x2, diag=False, **params):
-        x1_ = x1.div(self.period_length).mul(math.pi)
-        x2_ = x2.div(self.period_length).mul(math.pi)
-        diff = x1_.unsqueeze(-2) - x2_.unsqueeze(-3)
-        res = diff.sin().pow(2).sum(dim=-1).div(self.lengthscale).mul(-2.0).exp_()
+        # Pop this argument so that we can manually sum over dimensions
+        last_dim_is_batch = params.pop("last_dim_is_batch", False)
+        # Get lengthscale
+        lengthscale = self.lengthscale
+
+        x1_ = x1.div(self.period_length / math.pi)
+        x2_ = x2.div(self.period_length / math.pi)
+        # We are automatically overriding last_dim_is_batch here so that we can manually sum over dimensions.
+        diff = self.covar_dist(x1_, x2_, diag=diag, last_dim_is_batch=True, **params)
+
         if diag:
-            res = res.squeeze(0)
-        return res
+            lengthscale = lengthscale[..., 0, :, None]
+        else:
+            lengthscale = lengthscale[..., 0, :, None, None]
+        exp_term = diff.sin().pow(2.0).div(lengthscale).mul(-2.0)
+
+        if not last_dim_is_batch:
+            exp_term = exp_term.sum(dim=(-2 if diag else -3))
+
+        return exp_term.exp()


Test patch:
diff --git a/test/kernels/test_periodic_kernel.py b/test/kernels/test_periodic_kernel.py
--- a/test/kernels/test_periodic_kernel.py
+++ b/test/kernels/test_periodic_kernel.py
@@ -7,9 +7,16 @@
 
 from gpytorch.kernels import PeriodicKernel
 from gpytorch.priors import NormalPrior
+from gpytorch.test.base_kernel_test_case import BaseKernelTestCase
 
 
-class TestPeriodicKernel(unittest.TestCase):
+class TestPeriodicKernel(unittest.TestCase, BaseKernelTestCase):
+    def create_kernel_no_ard(self, **kwargs):
+        return PeriodicKernel(**kwargs)
+
+    def create_kernel_ard(self, num_dims, **kwargs):
+        return PeriodicKernel(ard_num_dims=num_dims, **kwargs)
+
     def test_computes_periodic_function(self):
         a = torch.tensor([4, 2, 8], dtype=torch.float).view(3, 1)
         b = torch.tensor([0, 2], dtype=torch.float).view(2, 1)