Repository: cornellius-gp/gpytorch

Issue:
[Bug] RBF Kernel is sometimes not positive definite
# 🐛 Bug

When I create a RBFKernel with 512 evenly spaced points between 500 and 501, and form the resulting kernel matrix, the matrix seems numerically positive definite (minimum eigenvalue is -1e-6). But, when I do the same with 513 points, the kernel matrix is very far from positive definite (minimum eigenvalue is -0.1). This behavior seems to crop up when using 513 points or more -- everything seems fine when using 512 or less; the results are also dependent on the range from which the datapoints are generated.

Is this a bug, or am I grossly misusing some feature?

## To reproduce

** Code snippet to reproduce **
```python
import numpy as np
import torch
import gpytorch

upper = 500
lower = 501
covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.RBFKernel())
covar_module.base_kernel.lengthscale = 1.0
covar_module.outputscale = 1.0


# Evaluate a RBF kernel on 512 evenly spaced points between 100 and
#  101. The resulting kernel matrix is numerically PSD (smallest eigenvalue
#  is -1e-6)
X = torch.linspace(lower, upper, 512)
covarX = covar_module(X, X).numpy()
print('Min eigenvalue with 512 points', np.linalg.eigvals(covarX).min())

# Evaluate a RBF kernel on 513 evenly spaced points between 100 and
#  101. The resulting kernel matrix is *not* numerically PSD (smallest eigenvalue
#  is -0.1)
X2 = torch.linspace(lower, upper, 513)
covarX2 = covar_module(X2, X2).numpy()
print('Min eigenvalue with 513 points', np.linalg.eigvals(covarX2).min())

```

** Stack trace/error message **
The output that I get from the above code is:
```
Min eigenvalue with 512 points -1.1961828e-06
Min eigenvalue with 513 points -0.18964253
```

## Expected Behavior

I would expect the minimum eigenvalue with 513 points to not be so negative.

## System information

**Please complete the following information:**
-  GPyTorch Version: 0.3.3
- PyTorch Version: 1.1.0 
- Computer OS: Ubuntu 18.04



Discussion:
This most likely has to do with the pairwise distance comparison, below size 512 this uses torch.cdist, and above a custom implementation: https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/kernels/kernel.py#L28-L29

Let's see what's going on there...
This seems to be a numerical precision issue when using floating point:

```
def custom_cdist(x1, x2):
    x1_norm = x1.pow(2).sum(dim=-1, keepdim=True)
    x1_pad = torch.ones_like(x1_norm)
    x2_norm = x2.pow(2).sum(dim=-1, keepdim=True)
    x2_pad = torch.ones_like(x2_norm)
    x1_ = torch.cat([-2. * x1, x1_norm, x1_pad], dim=-1)
    x2_ = torch.cat([x2, x2_pad, x2_norm], dim=-1)
    res = x1_.matmul(x2_.transpose(-2, -1))
    res.clamp_min_(0)
    return res

X2 = torch.linspace(lower, upper, 513).unsqueeze(-1)
X2d = X2.double()

diff = torch.max(torch.abs(torch.cdist(X2, X2).pow(2) - custom_cdist(X2, X2)))
diff_d = torch.max(torch.abs(torch.cdist(X2d, X2d).pow(2) - custom_cdist(X2d, X2d)))
```

diff: 0.0304
diff_d: 0.0

If you run your example in double you get 
```
Min eigenvalue with 512 points (-1.3547796728686771e-14+0j)
Min eigenvalue with 513 points -4.6523617484013634e-10`
```

This should of course work in float as well though...

@jacobrgardner
The numerical instability is caused by the very large values going in to the distance function (e.g., changing `lower` and `upper` to `0` and `1` will result in an identical kernel matrix as having them as `500` and `501`, but works.

Normally we rely on the user to do data normalization, but given that (1) it seems to outright fail when the feature values get this large and (2) the kernel matrix is identical, we could subtract the mean of the features like so:
```python
diff = torch.max(torch.abs(torch.cdist(X2, X2).pow(2) - custom_cdist(X2 - X2.mean(-2, keepdim=True), X2 - X2.mean(-2, keepdim=True))))
```
Then `diff == 0`

Resolving patch:
diff --git a/gpytorch/kernels/kernel.py b/gpytorch/kernels/kernel.py
--- a/gpytorch/kernels/kernel.py
+++ b/gpytorch/kernels/kernel.py
@@ -34,6 +34,10 @@ def _sq_dist(self, x1, x2, postprocess, x1_eq_x2=False):
             # cdist is a bit faster on small tensors. Check again after pytorch PR #20605
             res = torch.cdist(x1, x2).pow(2)
         else:
+            adjustment = x1.mean(-2, keepdim=True)
+            x1 = x1 - adjustment
+            x2 = x2 - adjustment  # x1 and x2 should be identical in all dims except -2 at this point
+
             # Compute squared distance matrix using quadratic expansion
             x1_norm = x1.pow(2).sum(dim=-1, keepdim=True)
             x1_pad = torch.ones_like(x1_norm)


Test patch: