Repository: cornellius-gp/gpytorch

Issue:
[Bug] get_fantasy_model does not work for SGPR with InducingPointKernel
# 🐛 Bug

Not sure if this should be considered a bug or a feature request, but gpytorch's implementation of SGPR using the InducingPointKernel kernel seems to not support get_fantasy_model.

## To reproduce
I am including the smallest mwe (or should I say mnwe) here. Note that I get the same behaviour by taking the [example tutorial for SGPR](https://gpytorch.readthedocs.io/en/latest/examples/05_Scalable_GP_Regression_Multidimensional/SGPR_Example_CUDA.html) and add a get_fantasy_model added at the end. I can post that too if required, but it is longer and might clutter the ticket.

**Code snippet to reproduce**
```python
import gpytorch
import torch
from gpytorch.kernels import ScaleKernel, RBFKernel, InducingPointKernel
from gpytorch.distributions import MultivariateNormal
from gpytorch.likelihoods import GaussianLikelihood
from gpytorch.means import ConstantMean

class GPRegressionModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super(GPRegressionModel, self).__init__(train_x, train_y, likelihood)
        self.mean_module = ConstantMean()
        self.base_covar_module = ScaleKernel(RBFKernel())
        self.covar_module = InducingPointKernel(self.base_covar_module, inducing_points=train_x[:500, :], likelihood=likelihood)

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return MultivariateNormal(mean_x, covar_x)

train_X = torch.randn((100,5)).to("cpu")
train_y = torch.randn((100)).to("cpu")

likelihood = GaussianLikelihood()

model = GPRegressionModel(train_X, train_y, likelihood)
model.train()
model.eval()

test_pred = model(torch.randn((1,5)).to("cpu"))

model = model.get_fantasy_model(torch.randn((1,5)).to("cpu"), torch.randn((1)).to("cpu"))
```

**Stack trace/error message**
```
Traceback (most recent call last):
  File "mwe_sgpr_fantasy.py", line 31, in <module>
    model = model.get_fantasy_model(torch.randn((1,5)).to("cpu"), torch.randn((1)).to("cpu"))
  File "/home/user/miniconda3/lib/python3.7/site-packages/gpytorch/models/exact_gp.py", line 173, in get_fantasy_model
    new_model = deepcopy(self)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 306, in _reconstruct
    value = deepcopy(value, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/home/user/miniconda3/lib/python3.7/site-packages/torch/tensor.py", line 23, in __deepcopy__
    raise RuntimeError("Only Tensors created explicitly by the user "
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment
```

## Expected Behavior

I would expect a fantasized model to be returned efficiently.

## System information

**Please complete the following information:**
- GPyTorch Version 0.3.3


- PyTorch Version 1.1.0
- Ubuntu 18.04

## Additional context
It seems that during the update, the `new_model = deepcopy(self)` tries to copy `self._inducing_inv_root` but detects that it is trainable by autograd and balks. I guess gpytorch made this design choice because of the goal of optimizing the inducing points as a hyperparameter, but as a tradeoff it does not allow for efficient updates.

So far I tried to replace the inducing points with a non-trainable version by setting `requires_grad` to `False`, but it seems to not help. I would guess that [any of these tensors multiplications](https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/kernels/inducing_point_kernel.py#L45-L47) in the implementation of `_inducing_inv_root` could end up reactivating autograd, and I am afraid that without more knowledge of gpytorch's internals patching them one-by-one might end up in a long whack-a-mole.



Discussion:

Resolving patch:
diff --git a/gpytorch/kernels/inducing_point_kernel.py b/gpytorch/kernels/inducing_point_kernel.py
--- a/gpytorch/kernels/inducing_point_kernel.py
+++ b/gpytorch/kernels/inducing_point_kernel.py
@@ -2,6 +2,7 @@
 
 import math
 import torch
+import copy
 from .kernel import Kernel
 from ..lazy import delazify, DiagLazyTensor, MatmulLazyTensor, RootLazyTensor, PsdSumLazyTensor
 from ..distributions import MultivariateNormal
@@ -96,3 +97,33 @@ def forward(self, x1, x2, diag=False, **kwargs):
 
     def num_outputs_per_input(self, x1, x2):
         return self.base_kernel.num_outputs_per_input(x1, x2)
+
+    def __deepcopy__(self, memo):
+        replace_inv_root = False
+        replace_kernel_mat = False
+
+        if hasattr(self, "_cached_kernel_inv_root"):
+            replace_inv_root = True
+            kernel_inv_root = self._cached_kernel_inv_root
+            self._cached_kernel_inv_root = None
+        if hasattr(self, "_cached_kernel_mat"):
+            replace_kernel_mat = True
+            kernel_mat = self._cached_kernel_mat
+            self._cached_kernel_mat = None
+
+        deepcopy_method = self.__deepcopy__
+        self.__deepcopy__ = None
+        cp = copy.deepcopy(self, memo)
+
+        self.__deepcopy__ = deepcopy_method
+        cp.__deepcopy__ = deepcopy_method
+
+        if replace_inv_root:
+            self._cached_kernel_inv_root = kernel_inv_root
+            cp._cached_kernel_inv_root = kernel_inv_root
+
+        if replace_kernel_mat:
+            self._cached_kernel_mat = kernel_mat
+            cp._cached_kernel_mat = kernel_mat
+
+        return cp


Test patch: