Repository: cornellius-gp/gpytorch

Issue:
Support predictions in prior mode
Right now, it's essentially impossible to use an `ExactGP` to get predictions from the prior, because (1) we require training data in exact GP models, (2) in `train` mode we ensure that the supplied data is the same as the training data, and (3) in `eval` mode we compute the predictive posterior.

Some applications (like BayesOpt) don't necessarily start with training data at all, so getting predictions from the prior can be useful for choosing initial points, for example.


Discussion:
This is a good point. By prior here I assume you mean a prior of the to-be-modeled function over the domain? Would this require a new kind of object?

In our applications, we often see that people would like to do "safe exploration", for which this would be quite useful (either using a prior based on some domain knowledge, or based on the posterior of a model from a related problem)
Yeah, I mean the prior over the to-be-modeled function. I've also had applications where it makes more sense to select points based on prior predictions to initialize BayesOpt than to choose randomly.

I don't think this will require a new object. **Technically** in code you can get prior predictions right now by creating a model using dummy tensors in `__init__` and then calling `forward` directly, which bypasses the check that we are calling on the training data in `model.train()` mode:

```python
likelihood = GaussianLikelihood(log_noise_bounds=(-5, 5))
model = ExactGPModel(torch.Tensor(0), torch.Tensor(0), likelihood)
prior_pred = model.forward(test_x)
prior_mean = prior_pred.mean()
prior_var = prior_pred.var()
prior_covar = prior_pred.covar().evaluate()
```
This interface is just obviously garbage.
> I've also had applications where it makes more sense to select points based on prior predictions to initialize BayesOpt than to choose randomly.

See also my comment on the design doc about a heuristic we use to select starting points for gradient-based optimization.

@bkarrer, @bletham


@jacobrgardner @Balandat maybe it would make sense for ExactGPModel to accept `None` for the training data arguments? This would be prior mode.
I think accepting `None` as training data is a good solution. That fits the intuition for what the prior is, so it's a good solution in my mind. Maybe `model.eval()` can throw an error if the data is None to prevent people from trying to get posterior predictions?
Or alternatively `model.eval()` can be what we *want* people to do to run in prior mode? The reason I mention it is because the other problem is that the model currently checks that the supplied data matches the training data in `train` mode, which obviously won't be true if the training data is `None`
@jacobrgardner I also think that always doing this check by default is a little much - maybe we should have some kind of contextmanager that we can use to turn if off if we don't need the check to be done?
I would suggest:

- `model.eval()` with `None` for training data - prior mode
- `model.eval()` with training data - posterior mode
- `model.train()` - training mode, requires training data

@Balandat I like the idea of using a context manager for doing the checks. We also make the check in `eval` mode if the evaluation data matches the training data, and I think this check would also be nice to turn off as well.

Resolving patch:
diff --git a/gpytorch/functions/__init__.py b/gpytorch/functions/__init__.py
--- a/gpytorch/functions/__init__.py
+++ b/gpytorch/functions/__init__.py
@@ -81,6 +81,9 @@ def exact_predictive_mean(full_covar, full_mean, train_labels, n_train, likeliho
     Returns:
     - (t) - the predictive posterior mean of the test points
     """
+    if not n_train:
+        return full_mean, None
+
     if not hasattr(full_covar, "exact_predictive_mean"):
         from ..lazy.non_lazy_variable import NonLazyVariable
 
@@ -102,6 +105,9 @@ def exact_predictive_covar(full_covar, n_train, likelihood, precomputed_cache=No
     Returns:
     - LazyVariable (t x t) - the predictive posterior covariance of the test points
     """
+    if not n_train:
+        return full_covar, None
+
     if not hasattr(full_covar, "exact_predictive_covar"):
         from ..lazy.non_lazy_variable import NonLazyVariable
 
diff --git a/gpytorch/models/exact_gp.py b/gpytorch/models/exact_gp.py
--- a/gpytorch/models/exact_gp.py
+++ b/gpytorch/models/exact_gp.py
@@ -5,33 +5,38 @@
 
 import warnings
 import torch
-from torch.autograd import Variable
 from ..module import Module
 from ..functions import exact_predictive_mean, exact_predictive_covar
 from ..random_variables import GaussianRandomVariable, MultitaskGaussianRandomVariable
 from ..likelihoods import GaussianLikelihood
+from .. import settings
 
 
 class ExactGP(Module):
     def __init__(self, train_inputs, train_targets, likelihood):
-        if torch.is_tensor(train_inputs):
+        if train_inputs is not None and torch.is_tensor(train_inputs):
             train_inputs = (train_inputs,)
-        if not all(torch.is_tensor(train_input) for train_input in train_inputs):
+        if train_inputs is not None and not all(torch.is_tensor(train_input) for train_input in train_inputs):
             raise RuntimeError("Train inputs must be a tensor, or a list/tuple of tensors")
         if not isinstance(likelihood, GaussianLikelihood):
             raise RuntimeError("ExactGP can only handle GaussianLikelihood")
 
         super(ExactGP, self).__init__()
-        self.train_inputs = tuple(tri.unsqueeze(-1) if tri.ndimension() == 1 else tri for tri in train_inputs)
-        self.train_targets = train_targets
+        if train_inputs is not None:
+            self.train_inputs = tuple(tri.unsqueeze(-1) if tri.ndimension() == 1 else tri for tri in train_inputs)
+            self.train_targets = train_targets
+        else:
+            self.train_inputs = None
+            self.train_targets = None
         self.likelihood = likelihood
 
         self.mean_cache = None
         self.covar_cache = None
 
     def _apply(self, fn):
-        self.train_inputs = tuple(fn(train_input) for train_input in self.train_inputs)
-        self.train_targets = fn(self.train_targets)
+        if self.train_inputs is not None:
+            self.train_inputs = tuple(fn(train_input) for train_input in self.train_inputs)
+            self.train_targets = fn(self.train_targets)
         return super(ExactGP, self)._apply(fn)
 
     def marginal_log_likelihood(self, likelihood, output, target, n_data=None):
@@ -73,47 +78,62 @@ def train(self, mode=True):
         return super(ExactGP, self).train(mode)
 
     def __call__(self, *args, **kwargs):
-        train_inputs = tuple(Variable(train_input) for train_input in self.train_inputs)
-        inputs = tuple(tri.unsqueeze(-1) if tri.ndimension() == 1 else tri for tri in args)
+        train_inputs = self.train_inputs
+        inputs = tuple(i.unsqueeze(-1) if i.ndimension() == 1 else i for i in args)
+
         # Training mode: optimizing
         if self.training:
-            if not all(torch.equal(train_input, input) for train_input, input in zip(train_inputs, inputs)):
-                raise RuntimeError("You must train on the training inputs!")
+            if self.train_inputs is None:
+                raise RuntimeError(
+                    "train_inputs, train_targets cannot be None in training mode. "
+                    "Call .eval() for prior predictions, or call .set_train_data() to add training data."
+                )
+            if settings.debug.on():
+                if not all(torch.equal(train_input, input) for train_input, input in zip(train_inputs, inputs)):
+                    raise RuntimeError("You must train on the training inputs!")
             return super(ExactGP, self).__call__(*inputs, **kwargs)
 
         # Posterior mode
         else:
-            if all(torch.equal(train_input, input) for train_input, input in zip(train_inputs, inputs)):
-                warnings.warn(
-                    "The input matches the stored training data. Did you forget to call model.train()?", UserWarning
-                )
+            if settings.debug.on() and self.train_inputs is not None:
+                if all(torch.equal(train_input, input) for train_input, input in zip(train_inputs, inputs)):
+                    warnings.warn(
+                        "The input matches the stored training data. Did you forget to call model.train()?", UserWarning
+                    )
 
             # Exact inference
-            full_inputs = tuple(
-                torch.cat([train_input, input], dim=-2) for train_input, input in zip(train_inputs, inputs)
-            )
+            if self.train_inputs is None:
+                full_inputs = args
+            else:
+                full_inputs = tuple(
+                    torch.cat([train_input, input], dim=-2) for train_input, input in zip(train_inputs, inputs)
+                )
             full_output = super(ExactGP, self).__call__(*full_inputs, **kwargs)
-            if not isinstance(full_output, GaussianRandomVariable):
-                raise RuntimeError("ExactGP.forward must return a GaussianRandomVariable")
+            if settings.debug().on():
+                if not isinstance(full_output, GaussianRandomVariable):
+                    raise RuntimeError("ExactGP.forward must return a GaussianRandomVariable")
             full_mean, full_covar = full_output.representation()
 
             n_tasks = 1
             if isinstance(full_output, MultitaskGaussianRandomVariable):
                 n_tasks = full_output.n_tasks
 
-            if self.train_targets.ndimension() == 2:
-                # Multitask
-                n_train = self.train_targets.size(0)
-                train_targets = self.train_targets.view(-1)
-                full_mean = full_mean.view(-1)
-            elif self.train_targets.ndimension() == 3:
-                # batch mode
-                n_train = self.train_targets.size(1)
-                train_targets = self.train_targets.view(self.train_targets.size(0), -1)
-                full_mean = full_mean.view(full_mean.size(0), -1)
-            else:
-                n_train = self.train_targets.size(-1)
-                train_targets = self.train_targets
+            n_train = 0
+            train_targets = None
+            if self.train_targets is not None:
+                if self.train_targets.ndimension() == 2:
+                    # Multitask
+                    full_mean = full_mean.view(-1)
+                    n_train = self.train_targets.size(0)
+                    train_targets = self.train_targets.view(-1)
+                elif self.train_targets.ndimension() == 3:
+                    # batch mode
+                    full_mean = full_mean.view(full_mean.size(0), -1)
+                    n_train = self.train_targets.size(1)
+                    train_targets = self.train_targets.view(self.train_targets.size(0), -1)
+                else:
+                    n_train = self.train_targets.size(-1)
+                    train_targets = self.train_targets
 
             predictive_mean, mean_cache = exact_predictive_mean(
                 full_covar=full_covar,
diff --git a/gpytorch/settings.py b/gpytorch/settings.py
--- a/gpytorch/settings.py
+++ b/gpytorch/settings.py
@@ -53,6 +53,17 @@ def __exit__(self, *args):
         return False
 
 
+class debug(_feature_flag):
+    """
+    Whether or not to perform "safety" checks on the supplied data.
+    (For example, that the correct training data is supplied in Exact GP training mode)
+    Pros: fewer data checks, fewer warning messages
+    Cons: possibility of supplying incorrect data, model accidentially in wrong mode
+    """
+
+    _state = True
+
+
 class max_cg_iterations(_value_context):
     """
     The maximum number of conjugate gradient iterations to perform (when computing
@@ -78,9 +89,8 @@ class max_root_decomposition_size(_value_context):
 class max_lanczos_iterations(max_root_decomposition_size):
     """
     The maximum number of Lanczos iterations to perform
-    This is used when 1) computing variance estiamtes 2) when drawing from MVNs,
-    or 3) for kernel multiplication
-    More values results in higher accuracy
+    This is used when 1) computing variance estiamtes 2) when drawing from
+    MVNs, or 3) for kernel multiplication More values results in higher accuracy
     Default: 100
 
     DEPRECATED: Use max_root_decomposition_size instead
@@ -113,6 +123,16 @@ class max_lanczos_quadrature_iterations(_value_context):
     _global_value = 15
 
 
+class memory_efficient(_feature_flag):
+    """
+    Whether or not to use Toeplitz math with gridded data, grid inducing point modules
+    Pros: memory efficient, faster on CPU
+    Cons: slower on GPUs with < 10000 inducing points
+    """
+
+    _state = False
+
+
 class num_likelihood_samples(_value_context):
     """
     The number of samples to draw from a latent GP when computing a likelihood
@@ -142,13 +162,3 @@ class use_toeplitz(_feature_flag):
     """
 
     _state = True
-
-
-class memory_efficient(_feature_flag):
-    """
-    Whether or not to use Toeplitz math with gridded data, grid inducing point modules
-    Pros: memory efficient, faster on CPU
-    Cons: slower on GPUs with < 10000 inducing points
-    """
-
-    _state = False


Test patch:
diff --git a/test/examples/test_multitask_gp_regression.py b/test/examples/test_multitask_gp_regression.py
--- a/test/examples/test_multitask_gp_regression.py
+++ b/test/examples/test_multitask_gp_regression.py
@@ -11,7 +11,6 @@
 import unittest
 import gpytorch
 from torch import optim
-from torch.autograd import Variable
 from gpytorch.kernels import RBFKernel, IndexKernel
 from gpytorch.likelihoods import GaussianLikelihood
 from gpytorch.means import ConstantMean
@@ -19,17 +18,17 @@
 from gpytorch.random_variables import GaussianRandomVariable
 
 # Simple training data: let's try to learn a sine function
-train_x = Variable(torch.linspace(0, 1, 100))
-y1_inds = Variable(torch.zeros(100).long())
-y2_inds = Variable(torch.ones(100).long())
-train_y1 = Variable(torch.sin(train_x.data * (2 * pi)))
-train_y2 = Variable(torch.cos(train_x.data * (2 * pi)))
+train_x = torch.linspace(0, 1, 100)
+y1_inds = torch.zeros(100).long()
+y2_inds = torch.ones(100).long()
+train_y1 = torch.sin(train_x * (2 * pi))
+train_y2 = torch.cos(train_x * (2 * pi))
 
-test_x = Variable(torch.linspace(0, 1, 51))
-y1_inds_test = Variable(torch.zeros(51).long())
-y2_inds_test = Variable(torch.ones(51).long())
-test_y1 = Variable(torch.sin(test_x.data * (2 * pi)))
-test_y2 = Variable(torch.cos(test_x.data * (2 * pi)))
+test_x = torch.linspace(0, 1, 51)
+y1_inds_test = torch.zeros(51).long()
+y2_inds_test = torch.ones(51).long()
+test_y1 = torch.sin(test_x * (2 * pi))
+test_y2 = torch.cos(test_x * (2 * pi))
 
 
 class MultitaskGPModel(gpytorch.models.ExactGP):
@@ -65,9 +64,7 @@ def tearDown(self):
     def test_multitask_gp_mean_abs_error(self):
         likelihood = GaussianLikelihood(log_noise_prior=SmoothedBoxPrior(-6, 6))
         gp_model = MultitaskGPModel(
-            (torch.cat([train_x.data, train_x.data]), torch.cat([y1_inds.data, y2_inds.data])),
-            torch.cat([train_y1.data, train_y2.data]),
-            likelihood,
+            (torch.cat([train_x, train_x]), torch.cat([y1_inds, y2_inds])), torch.cat([train_y1, train_y2]), likelihood
         )
         mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, gp_model)
 
@@ -98,12 +95,12 @@ def test_multitask_gp_mean_abs_error(self):
         test_preds_task_1 = likelihood(gp_model(test_x, y1_inds_test)).mean()
         mean_abs_error_task_1 = torch.mean(torch.abs(test_y1 - test_preds_task_1))
 
-        self.assertLess(mean_abs_error_task_1.data.squeeze().item(), 0.05)
+        self.assertLess(mean_abs_error_task_1.item(), 0.05)
 
         test_preds_task_2 = likelihood(gp_model(test_x, y2_inds_test)).mean()
         mean_abs_error_task_2 = torch.mean(torch.abs(test_y2 - test_preds_task_2))
 
-        self.assertLess(mean_abs_error_task_2.data.squeeze().item(), 0.05)
+        self.assertLess(mean_abs_error_task_2.item(), 0.05)
 
 
 if __name__ == "__main__":
diff --git a/test/examples/test_simple_gp_classification.py b/test/examples/test_simple_gp_classification.py
--- a/test/examples/test_simple_gp_classification.py
+++ b/test/examples/test_simple_gp_classification.py
@@ -11,7 +11,6 @@
 import unittest
 import gpytorch
 from torch import optim
-from torch.autograd import Variable
 from gpytorch.kernels import RBFKernel
 from gpytorch.lazy import ConstantMulLazyVariable
 from gpytorch.likelihoods import BernoulliLikelihood
@@ -21,8 +20,8 @@
 
 
 def train_data(cuda=False):
-    train_x = Variable(torch.linspace(0, 1, 10))
-    train_y = Variable(torch.sign(torch.cos(train_x.data * (4 * pi))))
+    train_x = torch.linspace(0, 1, 10)
+    train_y = torch.sign(torch.cos(train_x * (4 * pi)))
     if cuda:
         return train_x.cuda(), train_y.cuda()
     else:
@@ -65,7 +64,7 @@ def tearDown(self):
     def test_classification_error(self):
         train_x, train_y = train_data()
         likelihood = BernoulliLikelihood()
-        model = GPClassificationModel(train_x.data)
+        model = GPClassificationModel(train_x)
         mll = gpytorch.mlls.VariationalMarginalLogLikelihood(likelihood, model, n_data=len(train_y))
 
         # Find optimal model hyperparameters
@@ -94,13 +93,13 @@ def test_classification_error(self):
         likelihood.eval()
         test_preds = likelihood(model(train_x)).mean().ge(0.5).float().mul(2).sub(1).squeeze()
         mean_abs_error = torch.mean(torch.abs(train_y - test_preds) / 2)
-        assert mean_abs_error.data.squeeze().item() < 1e-5
+        assert mean_abs_error.item() < 1e-5
 
     def test_classification_fast_pred_var(self):
         with gpytorch.fast_pred_var():
             train_x, train_y = train_data()
             likelihood = BernoulliLikelihood()
-            model = GPClassificationModel(train_x.data)
+            model = GPClassificationModel(train_x)
             mll = gpytorch.mlls.VariationalMarginalLogLikelihood(likelihood, model, n_data=len(train_y))
 
             # Find optimal model hyperparameters
@@ -130,13 +129,13 @@ def test_classification_fast_pred_var(self):
             test_preds = likelihood(model(train_x)).mean().ge(0.5).float().mul(2).sub(1).squeeze()
 
             mean_abs_error = torch.mean(torch.abs(train_y - test_preds) / 2)
-            self.assertLess(mean_abs_error.data.squeeze().item(), 1e-5)
+            self.assertLess(mean_abs_error.item(), 1e-5)
 
     def test_classification_error_cuda(self):
         if torch.cuda.is_available():
             train_x, train_y = train_data(cuda=True)
             likelihood = BernoulliLikelihood().cuda()
-            model = GPClassificationModel(train_x.data).cuda()
+            model = GPClassificationModel(train_x).cuda()
             mll = gpytorch.mlls.VariationalMarginalLogLikelihood(likelihood, model, n_data=len(train_y))
 
             # Find optimal model hyperparameters
@@ -163,7 +162,7 @@ def test_classification_error_cuda(self):
             model.eval()
             test_preds = likelihood(model(train_x)).mean().ge(0.5).float().mul(2).sub(1).squeeze()
             mean_abs_error = torch.mean(torch.abs(train_y - test_preds) / 2)
-            self.assertLess(mean_abs_error.data.squeeze().item(), 1e-5)
+            self.assertLess(mean_abs_error.item(), 1e-5)
 
 
 if __name__ == "__main__":
diff --git a/test/examples/test_simple_gp_regression.py b/test/examples/test_simple_gp_regression.py
--- a/test/examples/test_simple_gp_regression.py
+++ b/test/examples/test_simple_gp_regression.py
@@ -11,7 +11,6 @@
 import unittest
 import gpytorch
 from torch import optim
-from torch.autograd import Variable
 from gpytorch.kernels import RBFKernel
 from gpytorch.likelihoods import GaussianLikelihood
 from gpytorch.means import ConstantMean
@@ -20,11 +19,11 @@
 
 
 # Simple training data: let's try to learn a sine function
-train_x = Variable(torch.linspace(0, 1, 11))
-train_y = Variable(torch.sin(train_x.data * (2 * pi)))
+train_x = torch.linspace(0, 1, 11)
+train_y = torch.sin(train_x * (2 * pi))
 
-test_x = Variable(torch.linspace(0, 1, 51))
-test_y = Variable(torch.sin(test_x.data * (2 * pi)))
+test_x = torch.linspace(0, 1, 51)
+test_y = torch.sin(test_x * (2 * pi))
 
 
 class ExactGPModel(gpytorch.models.ExactGP):
@@ -54,12 +53,36 @@ def tearDown(self):
         if hasattr(self, "rng_state"):
             torch.set_rng_state(self.rng_state)
 
+    def test_prior(self):
+        # We're manually going to set the hyperparameters to be ridiculous
+        likelihood = GaussianLikelihood(
+            log_noise_prior=SmoothedBoxPrior(exp(-3), exp(3), sigma=0.1, log_transform=True)
+        )
+        gp_model = ExactGPModel(None, None, likelihood)
+        # Update lengthscale prior to accommodate extreme parameters
+        gp_model.covar_module.set_parameter_priors(
+            log_lengthscale=SmoothedBoxPrior(exp(-10), exp(10), sigma=0.5, log_transform=True)
+        )
+        gp_model.mean_module.initialize(constant=1.5)
+        gp_model.covar_module.initialize(log_lengthscale=0)
+        likelihood.initialize(log_noise=0)
+
+        # Compute posterior distribution
+        gp_model.eval()
+        likelihood.eval()
+
+        # The model should predict in prior mode
+        function_predictions = likelihood(gp_model(train_x))
+
+        self.assertLess(torch.norm(function_predictions.mean() - 1.5), 1e-3)
+        self.assertLess(torch.norm(function_predictions.var() - 2), 1e-3)
+
     def test_posterior_latent_gp_and_likelihood_without_optimization(self):
         # We're manually going to set the hyperparameters to be ridiculous
         likelihood = GaussianLikelihood(
             log_noise_prior=SmoothedBoxPrior(exp(-3), exp(3), sigma=0.1, log_transform=True)
         )
-        gp_model = ExactGPModel(train_x.data, train_y.data, likelihood)
+        gp_model = ExactGPModel(train_x, train_y, likelihood)
         # Update lengthscale prior to accommodate extreme parameters
         gp_model.covar_module.set_parameter_priors(
             log_lengthscale=SmoothedBoxPrior(exp(-10), exp(10), sigma=0.5, log_transform=True)
@@ -73,23 +96,24 @@ def test_posterior_latent_gp_and_likelihood_without_optimization(self):
 
         # Let's see how our model does, conditioned with weird hyperparams
         # The posterior should fit all the data
-        function_predictions = likelihood(gp_model(train_x))
+        with gpytorch.settings.debug(False):
+            function_predictions = likelihood(gp_model(train_x))
 
-        self.assertLess(torch.norm(function_predictions.mean().data - train_y.data), 1e-3)
-        self.assertLess(torch.norm(function_predictions.var().data), 1e-3)
+        self.assertLess(torch.norm(function_predictions.mean() - train_y), 1e-3)
+        self.assertLess(torch.norm(function_predictions.var()), 1e-3)
 
         # It shouldn't fit much else though
-        test_function_predictions = gp_model(Variable(torch.Tensor([1.1])))
+        test_function_predictions = gp_model(torch.Tensor([1.1]))
 
-        self.assertLess(torch.norm(test_function_predictions.mean().data - 0), 1e-4)
-        self.assertLess(torch.norm(test_function_predictions.var().data - 1), 1e-4)
+        self.assertLess(torch.norm(test_function_predictions.mean() - 0), 1e-4)
+        self.assertLess(torch.norm(test_function_predictions.var() - 1), 1e-4)
 
     def test_posterior_latent_gp_and_likelihood_with_optimization(self):
         # We're manually going to set the hyperparameters to something they shouldn't be
         likelihood = GaussianLikelihood(
             log_noise_prior=SmoothedBoxPrior(exp(-3), exp(3), sigma=0.1, log_transform=True)
         )
-        gp_model = ExactGPModel(train_x.data, train_y.data, likelihood)
+        gp_model = ExactGPModel(train_x, train_y, likelihood)
         mll = gpytorch.ExactMarginalLogLikelihood(likelihood, gp_model)
         gp_model.covar_module.initialize(log_lengthscale=1)
         gp_model.mean_module.initialize(constant=0)
@@ -102,7 +126,8 @@ def test_posterior_latent_gp_and_likelihood_with_optimization(self):
         optimizer.n_iter = 0
         for _ in range(50):
             optimizer.zero_grad()
-            output = gp_model(train_x)
+            with gpytorch.settings.debug(False):
+                output = gp_model(train_x)
             loss = -mll(output, train_y)
             loss.backward()
             optimizer.n_iter += 1
@@ -122,16 +147,16 @@ def test_posterior_latent_gp_and_likelihood_with_optimization(self):
         test_function_predictions = likelihood(gp_model(test_x))
         mean_abs_error = torch.mean(torch.abs(test_y - test_function_predictions.mean()))
 
-        self.assertLess(mean_abs_error.data.squeeze().item(), 0.05)
+        self.assertLess(mean_abs_error.item(), 0.05)
 
     def test_posterior_latent_gp_and_likelihood_fast_pred_var(self):
-        with gpytorch.fast_pred_var():
+        with gpytorch.fast_pred_var(), gpytorch.settings.debug(False):
             # We're manually going to set the hyperparameters to
             # something they shouldn't be
             likelihood = GaussianLikelihood(
                 log_noise_prior=SmoothedBoxPrior(exp(-3), exp(3), sigma=0.1, log_transform=True)
             )
-            gp_model = ExactGPModel(train_x.data, train_y.data, likelihood)
+            gp_model = ExactGPModel(train_x, train_y, likelihood)
             mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, gp_model)
             gp_model.covar_module.initialize(log_lengthscale=1)
             gp_model.mean_module.initialize(constant=0)
@@ -172,7 +197,7 @@ def test_posterior_latent_gp_and_likelihood_fast_pred_var(self):
             noise = likelihood.log_noise.exp()
             var_diff = (test_function_predictions.var() - noise).abs()
 
-            self.assertLess(torch.max(var_diff.data / noise.data), 0.05)
+            self.assertLess(torch.max(var_diff / noise), 0.05)
 
     def test_posterior_latent_gp_and_likelihood_with_optimization_cuda(self):
         if torch.cuda.is_available():
@@ -181,7 +206,7 @@ def test_posterior_latent_gp_and_likelihood_with_optimization_cuda(self):
             likelihood = GaussianLikelihood(
                 log_noise_prior=SmoothedBoxPrior(exp(-3), exp(3), sigma=0.1, log_transform=True)
             ).cuda()
-            gp_model = ExactGPModel(train_x.data.cuda(), train_y.data.cuda(), likelihood).cuda()
+            gp_model = ExactGPModel(train_x.cuda(), train_y.cuda(), likelihood).cuda()
             mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, gp_model)
             gp_model.covar_module.initialize(log_lengthscale=1)
             gp_model.mean_module.initialize(constant=0)
@@ -214,7 +239,7 @@ def test_posterior_latent_gp_and_likelihood_with_optimization_cuda(self):
             test_function_predictions = likelihood(gp_model(test_x.cuda()))
             mean_abs_error = torch.mean(torch.abs(test_y.cuda() - test_function_predictions.mean()))
 
-            self.assertLess(mean_abs_error.data.squeeze().item(), 0.05)
+            self.assertLess(mean_abs_error.item(), 0.05)
 
 
 if __name__ == "__main__":