Repository: cornellius-gp/gpytorch

Issue:
[Bug] Invalid index in gather during call to mll in training loop
# 🐛 Bug

I followed the docs related to DKL exactly and got the following error during the training loop when calculating the loss with the marginal likelihood function

** Stack trace/error message **
```
Traceback (most recent call last):
  File "gp.py", line 24, in <module>
    exact.train()
  File "/st2/jeff/real_estate/models/gaussian_processes/exact.py", line 102, in train
    loss = -mll(output, train_y).sum()
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch/module.py", line 22, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch/mlls/exact_marginal_log_likelihood.py", line 27, in forward
    res = output.log_prob(target)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch/distributions/multivariate_normal.py", line 128, in log_prob
    inv_quad, logdet = covar.inv_quad_logdet(inv_quad_rhs=diff.unsqueeze(-1), logdet=True)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch/lazy/batch_repeat_lazy_tensor.py", line 242, in inv_quad_logdet
    inv_quad_rhs, logdet, reduce_inv_quad=False
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch/lazy/lazy_tensor.py", line 1052, in inv_quad_logdet
    *args,
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch/functions/_inv_quad_log_det.py", line 63, in forward
    preconditioner, precond_lt, logdet_correction = lazy_tsr._preconditioner()
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch/lazy/added_diag_lazy_tensor.py", line 59, in _preconditioner
    self._piv_chol_self = pivoted_cholesky.pivoted_cholesky(self._lazy_tensor, max_iter)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch/utils/pivoted_cholesky.py", line 19, in pivoted_cholesky
    matrix_diag = matrix._approx_diag()
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch/lazy/interpolated_lazy_tensor.py", line 90, in _approx_diag
    left_res = left_interp(self.left_interp_indices, self.left_interp_values, base_diag_root.unsqueeze(-1))
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch/utils/interpolation.py", line 187, in left_interp
    res = rhs_expanded.gather(-3, interp_indices_expanded).mul(interp_values_expanded)
RuntimeError: Invalid index in gather at /tmp/pip-req-build-58y_cjjl/aten/src/TH/generic/THTensorEvenMoreMath.cpp:472
loss: 57158.71 med: 0.30, minmax: 0.30 0.30 noise: 0.56: : 0it [00:08, ?it/s]
```

## Expected Behavior

I am left unsure of what is causing the error and how to go about fixing it because it is initially successful in iterating and calculating the loss and then it crashes. The sizing of tensors must be correct, but there is must be some numerical instability and I am unsure about where to look for it.

## System information

**Please complete the following information:**
gpytorch version: `0.3.6`
torch version: `1.2.0
Ubuntu 18.04



Discussion:
@deltaskelta can you provide a more complete example? I don't know how to reproduce.
Can you also try on the latest version of gpytorch (off of the `master` branch on Github) and PyTorch 1.3.1?
@gpleiss thanks for your response. I installed the most recent pytorch and gpytorch from the master branch and I still had the issue. I had time to create a small reproduction repo (https://github.com/deltaskelta/gpytorch-955) with a minimal dataset. You can run it by doing...

```
python dkl.py
```

I printed the input tensor to make sure no weird values were showing up. If you let it run for a minute or so it should fail with the above error very quickly
Thanks so much for the repo! I've reproduced the error - and will look into it.

Resolving patch:
diff --git a/gpytorch/utils/interpolation.py b/gpytorch/utils/interpolation.py
--- a/gpytorch/utils/interpolation.py
+++ b/gpytorch/utils/interpolation.py
@@ -39,7 +39,7 @@ def _cubic_interpolation_kernel(self, scaled_grid_dist):
         res = res + (((-0.5 * U + 2.5).mul(U) - 4).mul(U) + 2) * U_ge_1_le_2
         return res
 
-    def interpolate(self, x_grid: List[torch.Tensor], x_target: torch.Tensor, interp_points=range(-2, 2)):
+    def interpolate(self, x_grid: List[torch.Tensor], x_target: torch.Tensor, interp_points=range(-2, 2), eps=1e-10):
         if torch.is_tensor(x_grid):
             x_grid = convert_legacy_grid(x_grid)
         num_target_points = x_target.size(0)
@@ -99,7 +99,7 @@ def interpolate(self, x_grid: List[torch.Tensor], x_target: torch.Tensor, interp
 
         for i in range(num_dim):
             num_grid_points = x_grid[i].size(0)
-            grid_delta = x_grid[i][1] - x_grid[i][0]
+            grid_delta = (x_grid[i][1] - x_grid[i][0]).clamp_min_(eps)
             # left-bounding grid point in index space
             lower_grid_pt_idxs = torch.floor((x_target[:, i] - x_grid[i][0]) / grid_delta)
             # distance from that left-bounding grid point, again in index space


Test patch: