Repository: cornellius-gp/gpytorch

Issue:
[Bug] IndexError when GridInterpolationKernel wraps a PeriodicKernel
# 🐛 Bug

I tried to adapt the [KISS-GP for 1D data tutorial](https://gpytorch.readthedocs.io/en/latest/examples/02_Scalable_Exact_GPs/KISSGP_Regression.html#KISS-GP-for-1D-Data) to use a `PeriodicKernel` instead of an `RBFKernel`.

However, trying to evaluate the MLL fails with an `IndexError: Dimension out of range (expected to be in range of [-1, 0], but got -2)`, thrown from [grid_kernel.py: 133](https://github.com/cornellius-gp/gpytorch/blob/0317b121ebaaa921a7851a6af4f2219ff18eeaf0/gpytorch/kernels/grid_kernel.py#L133).

It seems to me this can only be a bug, as an RBF kernel and a Periodic kernel are really not very different semantically?

## To reproduce

** Code snippet to reproduce **

```python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Tue Jan  7 13:52:48 2020

@author: val
"""


import math
import torch
import gpytorch

#%matplotlib inline

train_x = torch.linspace(0, 1, 1000)
train_y = torch.sin(train_x * (4 * math.pi) + torch.randn(train_x.size()) * 0.2)

class GPRegressionModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super(GPRegressionModel, self).__init__(train_x, train_y, likelihood)

        # SKI requires a grid size hyperparameter. This util can help with that. Here we are using a grid that has the same number of points as the training data (a ratio of 1.0). Performance can be sensitive to this parameter, so you may want to adjust it for your own problem on a validation set.
        grid_size = gpytorch.utils.grid.choose_grid_size(train_x,1.0)

        self.mean_module = gpytorch.means.ConstantMean()
        self.covar_module = gpytorch.kernels.ScaleKernel(
            gpytorch.kernels.GridInterpolationKernel(    
                ## NOTE the only difference from the 1D KISS-GP tutorial is that the RBFKernel got replaced with a PeriodicKernel()
                gpytorch.kernels.PeriodicKernel(), #gpytorch.kernels.RBFKernel(),
                grid_size=grid_size, num_dims=1
            )
        )
        #self.covar_module = gpytorch.kernels.PeriodicKernel() 

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)


likelihood = gpytorch.likelihoods.GaussianLikelihood()
model = GPRegressionModel(train_x, train_y, likelihood)

training_iterations = 1

# Find optimal model hyperparameters
model.train()
likelihood.train()

# Use the adam optimizer
optimizer = torch.optim.Adam([
    {'params': model.parameters()},  # Includes GaussianLikelihood parameters
], lr=0.1)

# "Loss" for GPs - the marginal log likelihood
mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model)

for i in range(training_iterations):
    optimizer.zero_grad()
    output = model(train_x)
    loss = -mll(output, train_y) ## NOTE fails here.
    loss.backward()
    optimizer.step()
```

** Stack trace/error message **

```
Traceback (most recent call last):

  File "<ipython-input-13-effc04c4ab77>", line 61, in <module>
    loss = -mll(output, train_y) ## NOTE fails here.

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/module.py", line 24, in __call__
    outputs = self.forward(*inputs, **kwargs)

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/mlls/exact_marginal_log_likelihood.py", line 51, in forward
    res = output.log_prob(target)

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/distributions/multivariate_normal.py", line 135, in log_prob
    inv_quad, logdet = covar.inv_quad_logdet(inv_quad_rhs=diff.unsqueeze(-1), logdet=True)

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/lazy/lazy_tensor.py", line 1038, in inv_quad_logdet
    args = self.representation()

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/lazy/lazy_tensor.py", line 1268, in representation
    representation += list(arg.representation())

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/lazy/lazy_evaluated_kernel_tensor.py", line 311, in representation
    return self.evaluate_kernel().representation()

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/utils/memoize.py", line 34, in g
    add_to_cache(self, cache_name, method(self, *args, **kwargs))

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/lazy/lazy_evaluated_kernel_tensor.py", line 278, in evaluate_kernel
    res = self.kernel(x1, x2, diag=False, last_dim_is_batch=self.last_dim_is_batch, **self.params)

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/kernels/kernel.py", line 395, in __call__
    res = super(Kernel, self).__call__(x1_, x2_, last_dim_is_batch=last_dim_is_batch, **params)

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/module.py", line 24, in __call__
    outputs = self.forward(*inputs, **kwargs)

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/kernels/scale_kernel.py", line 90, in forward
    orig_output = self.base_kernel.forward(x1, x2, diag=diag, last_dim_is_batch=last_dim_is_batch, **params)

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/kernels/grid_interpolation_kernel.py", line 177, in forward
    base_lazy_tsr = lazify(self._inducing_forward(last_dim_is_batch=last_dim_is_batch, **params))

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/kernels/grid_interpolation_kernel.py", line 143, in _inducing_forward
    return super().forward(self.grid, self.grid, last_dim_is_batch=last_dim_is_batch, **params)

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/kernels/grid_kernel.py", line 133, in forward
    covars = [ToeplitzLazyTensor(c.squeeze(-2)) for c in covars]

  File "/Users/val/opt/anaconda3/lib/python3.7/site-packages/gpytorch/kernels/grid_kernel.py", line 133, in <listcomp>
    covars = [ToeplitzLazyTensor(c.squeeze(-2)) for c in covars]

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got -2)
```

## Expected Behavior

I expected the training loop to terminate successfully, without throwing errors.


## System information

**Please complete the following information:**
- GPyTorch Version: 1.0.0
- PyTorch Version: 1.3.1
- Computer OS macOS High Sierra 10.13.3




Discussion:
@vvvvalvalval Hmm, this does look like a bug. I'll take a look at this. 

Resolving patch:
diff --git a/gpytorch/kernels/periodic_kernel.py b/gpytorch/kernels/periodic_kernel.py
--- a/gpytorch/kernels/periodic_kernel.py
+++ b/gpytorch/kernels/periodic_kernel.py
@@ -111,6 +111,6 @@ def forward(self, x1, x2, diag=False, **params):
         x2_ = x2.div(self.period_length)
         diff = self.covar_dist(x1_, x2_, diag=diag, **params)
         res = torch.sin(diff.mul(math.pi)).pow(2).mul(-2 / self.lengthscale).exp_()
-        if diff.ndimension() == 2 or diag:
+        if diag:
             res = res.squeeze(0)
         return res


Test patch: