internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: cornellius-gp/gpytorch

Issue:
Where is `fast_pred_var` moving?
I received the following warning when using `fast_pred_var`:

```
/cluster/nhunt/anaconda/envs/bayes_opt/lib/python3.7/site-packages/gpytorch/beta_features.py:17:
DeprecationWarning: `gpytorch.settings.fast_pred_var` has moved to `gpytorch.settings.fast_pred_var`.
```

It seems that I'm being warned that `fast_pred_var` has moved to its current location. Was there a typo in the warning about how we should be using this setting now?

```bash
$ pip list | grep gpytorch
gpytorch           0.2.1
```


Discussion:
@gpleiss This seems like a problem with the autogenerated warning.

@neighthan `fast_pred_var` now lives in `settings` -- the location you are seeing it at is the correct location.
Hmmm yeah I'll see if I can figure out what the issue is. The correct context manager is `with gpytorch.settings.fast_pred_var`

(It used to be either `gpytorch.beta_features.fast_pred_var` an also as `gpytorch.fast_pred_var`, but these are both deprecated)
Ah, I see - I was actually using `gpytorch.fast_pred_var`, so it was correct that I received the deprecation warning. It was just confusing that in the warning it said `gpytorch.settings.fast_pred_var` both times (instead of `gpytorch.fast_pred_var` has moved to `gpytorch.settings.fast_pred_var` or `gpytorch.beta_features.fast_pred_var` has moved to `gpytorch.settings.fast_pred_var`, depending on how it was accessed). Thanks for clearing that up! I'll leave the issue open for fixing the warning message.

Resolving patch:
diff --git a/gpytorch/__init__.py b/gpytorch/__init__.py
--- a/gpytorch/__init__.py
+++ b/gpytorch/__init__.py
@@ -37,9 +37,6 @@
 
 __version__ = "0.2.1"
 
-# Old deprecated stuff
-fast_pred_var = beta_features._moved_beta_feature(settings.fast_pred_var, "gpytorch.settings.fast_pred_var")
-
 __all__ = [
     # Submodules
     "distributions",
@@ -77,7 +74,6 @@
     # Other
     "__version__",
     # Deprecated
-    "fast_pred_var",
     "inv_quad_log_det",
     "log_det",
 ]
diff --git a/gpytorch/beta_features.py b/gpytorch/beta_features.py
--- a/gpytorch/beta_features.py
+++ b/gpytorch/beta_features.py
@@ -2,8 +2,6 @@
 
 import warnings
 from .settings import _feature_flag, _value_context
-from .settings import fast_pred_var as _fast_pred_var
-from .settings import fast_pred_samples as _fast_pred_samples
 
 
 class _moved_beta_feature(object):
@@ -22,10 +20,6 @@ def __getattr__(self, name):
         return getattr(self.new_cls, name)
 
 
-fast_pred_var = _moved_beta_feature(_fast_pred_var)
-fast_pred_samples = _moved_beta_feature(_fast_pred_samples)
-
-
 class checkpoint_kernel(_value_context):
     """
     Should the kernel be computed in chunks with checkpointing or not? (Default, no)
@@ -61,4 +55,4 @@ class default_preconditioner(_feature_flag):
     pass
 
 
-__all__ = ["fast_pred_var", "fast_pred_samples", "diagonal_correction", "default_preconditioner"]
+__all__ = ["checkpoint_kernel", "diagonal_correction", "default_preconditioner"]


Test patch:
Repository: cornellius-gp/gpytorch

Issue:
[Bug] TypeError when calling `backward` on `gpytorch.functions.logdet` 
# 🐛 Bug

I think there's a missing `list()` in https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/functions/_inv_quad_log_det.py#L221. I'm not super familiar with gpytorch internals so hopefully this is correct -- if so, happy to contribute the one-liner fix. 

## To reproduce

** Code snippet to reproduce **
```python
### works (I'm guessing something dispatches elsewhere for small matrices?)
import torch
from torch.autograd import backward
import gpytorch
from gpytorch.functions import logdet, inv_matmul
n = 100
inp = torch.arange(n, dtype=torch.float)
kern = gpytorch.kernels.RBFKernel()(inp)
ld = logdet(kern)
backward(ld)

### doesn't work
import torch
from torch.autograd import backward
import gpytorch
from gpytorch.functions import logdet, inv_matmul
n = 1000
inp = torch.arange(n, dtype=torch.float)
kern = gpytorch.kernels.RBFKernel()(inp)
ld = logdet(kern)
backward(ld)
```

** Stack trace/error message **
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-46-593fbced29ac> in <module>()
      3 kern = gpytorch.kernels.RBFKernel()(inp)
      4 ld = logdet(kern)
----> 5 backward(ld)

<PATH SNIPPED>/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     91     Variable._execution_engine.run_backward(
     92         tensors, grad_tensors, retain_graph, create_graph,
---> 93         allow_unreachable=True)  # allow_unreachable flag
     94 
     95 

<PATH SNIPPED>/lib/python3.7/site-packages/torch/autograd/function.py in apply(self, *args)
     75 
     76     def apply(self, *args):
---> 77         return self._forward_cls.backward(self, *args)
     78 
     79 

<PATH SNIPPED>lib/python3.7/site-packages/gpytorch/functions/_inv_quad_log_det.py in backward(ctx, inv_quad_grad_output, logdet_grad_output)
    221             res = matrix_arg_grads
    222 
--> 223         return tuple([None] * 9 + res)

TypeError: can only concatenate list (not "tuple") to list
```

## Expected Behavior

No error. 

## System information

**Please complete the following information:**
- GPyTorch version: 0.3.2 <!-- GPyTorch Version (run `print(gpytorch.__version__)` -->
- pytorch version 1.1.0. 
- Mac OSX. 




Discussion:
Ah you’re totally right! Good catch. A one-line fix would be great!

Resolving patch:
diff --git a/gpytorch/functions/_inv_quad_log_det.py b/gpytorch/functions/_inv_quad_log_det.py
--- a/gpytorch/functions/_inv_quad_log_det.py
+++ b/gpytorch/functions/_inv_quad_log_det.py
@@ -218,6 +218,6 @@ def backward(ctx, inv_quad_grad_output, logdet_grad_output):
         if ctx.inv_quad:
             res = [inv_quad_rhs_grad] + list(matrix_arg_grads)
         else:
-            res = matrix_arg_grads
+            res = list(matrix_arg_grads)
 
         return tuple([None] * 9 + res)


Test patch:
Repository: cornellius-gp/gpytorch

Issue:
[Bug] Cannot serialize/deserialize SmoothedBoxPrior when some args are broadcast
# 🐛 Bug

It seems like `SmoothedBoxPrior` for >1d doesn't work with serialization/deserialization when only some args are broadcast. 

## To reproduce

```python
import torch
import gpytorch
pr = gpytorch.priors.SmoothedBoxPrior(torch.zeros(2), torch.ones(2))
pr.load_state_dict(pr.state_dict())
```

** Stack trace/error message **
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-6b4b2e881beb> in <module>
      2 import gpytorch
      3 pr = gpytorch.priors.SmoothedBoxPrior(torch.zeros(2), torch.ones(2))
----> 4 pr.load_state_dict(pr.state_dict())

<...PATH..>/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
    877         if len(error_msgs) > 0:
    878             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
--> 879                                self.__class__.__name__, "\n\t".join(error_msgs)))
    880         return _IncompatibleKeys(missing_keys, unexpected_keys)
    881 

RuntimeError: Error(s) in loading state_dict for SmoothedBoxPrior:
	While copying the parameter named "sigma", whose dimensions in the model are torch.Size([2]) and whose dimensions in the checkpoint are torch.Size([2]), an exception occured : ('unsupported operation: more than one element of the written-to tensor refers to a single memory location. Please clone() the tensor before performing the operation.',).

```

Note that `SmoothedBoxPrior(a=torch.zeros(2), b=torch.ones(2), sigma=torch.ones(2)*0.01)` succeeds, as does `gpytorch.priors.GammaPrior(torch.ones(2),1)`.

## Expected Behavior

Successful load. 

## System information

**Please complete the following information:**
- gpytorch version: 1.1.1
- pytorch version: 1.5.0
- OS: tested on Centos and Mac OSX. 





Discussion:
Huh this is curious. We'll take a look.
(P.S. thanks for the super simple code example!)

Resolving patch:
diff --git a/gpytorch/priors/smoothed_box_prior.py b/gpytorch/priors/smoothed_box_prior.py
--- a/gpytorch/priors/smoothed_box_prior.py
+++ b/gpytorch/priors/smoothed_box_prior.py
@@ -47,7 +47,7 @@ def __init__(self, a, b, sigma=0.01, validate_args=False, transform=None):
         del self.a, self.b, self.sigma
         self.register_buffer("a", _a)
         self.register_buffer("b", _b)
-        self.register_buffer("sigma", _sigma)
+        self.register_buffer("sigma", _sigma.clone())
         self.tails = NormalPrior(torch.zeros_like(_a), _sigma, validate_args=validate_args)
         self._transform = transform
 


Test patch:
Repository: cornellius-gp/gpytorch

Issue:
[Bug] RFFPredictionStrategy should allow for turning off predictive variances
See here: https://github.com/cornellius-gp/gpytorch/blob/3bffd226773777a652267ec187459e52f85757a0/gpytorch/models/exact_prediction_strategies.py#L650

`gpytorch.settings.skip_posterior_variances()` fails to work under this prediction strategy. Potentially problematic because I'm noticing that the explicit cholesky in the `covar_cache` occasionally fails when predicting with a lot of data points and not as many features. 

Fix is to include these lines in the call as well:

https://github.com/cornellius-gp/gpytorch/blob/3bffd226773777a652267ec187459e52f85757a0/gpytorch/models/exact_prediction_strategies.py#L359



Discussion:

Resolving patch:
diff --git a/gpytorch/models/exact_prediction_strategies.py b/gpytorch/models/exact_prediction_strategies.py
--- a/gpytorch/models/exact_prediction_strategies.py
+++ b/gpytorch/models/exact_prediction_strategies.py
@@ -648,6 +648,9 @@ def exact_prediction(self, joint_mean, joint_covar):
         )
 
     def exact_predictive_covar(self, test_test_covar, test_train_covar):
+        if settings.skip_posterior_variances.on():
+            return ZeroLazyTensor(*test_test_covar.size())
+
         if isinstance(test_test_covar, ConstantMulLazyTensor):
             constant = test_test_covar.expanded_constant
             test_test_covar = test_test_covar.base_lazy_tensor


Test patch:
Repository: cornellius-gp/gpytorch

Issue:
[Bug] Typo in doc - Matern Kernel
# 🐛 Bug

Should the documentation here https://docs.gpytorch.ai/en/v1.1.1/kernels.html#maternkernel have $ \Theta^{-2}$ in the definition of `d` like the other kernels (example RBF)? 



Discussion:
That does seem like a typo: https://github.com/cornellius-gp/gpytorch/blob/169b3ad9d85a6e4dfd4a0456dcaefc27c004384c/gpytorch/kernels/matern_kernel.py#L100
Is d defined a different way then? Maybe it should be $d=\sqrt( (x_1 - x_2)^T (x_1-x_2))$ when $\theta=1$. Otherwise the units don't seem to add up. 
Also, I think that equation you are pointing to seems to suggest that when I take that x1_ and (x2_) and put it in the formula for d in the doc, then there would be a lengthscale^2 in the denominator. 
`d` should be the distance after rescaling, so $d = \sqrt((x_1-x_2)^\top \Theta^{-2} (x_1 - x_2))$

Resolving patch:
diff --git a/gpytorch/kernels/matern_kernel.py b/gpytorch/kernels/matern_kernel.py
--- a/gpytorch/kernels/matern_kernel.py
+++ b/gpytorch/kernels/matern_kernel.py
@@ -23,7 +23,7 @@ class MaternKernel(Kernel):
 
     where
 
-    * :math:`d = (\mathbf{x_1} - \mathbf{x_2})^\top \Theta^{-1} (\mathbf{x_1} - \mathbf{x_2})`
+    * :math:`d = (\mathbf{x_1} - \mathbf{x_2})^\top \Theta^{-2} (\mathbf{x_1} - \mathbf{x_2})`
       is the distance between
       :math:`x_1` and :math:`x_2` scaled by the :attr:`lengthscale` parameter :math:`\Theta`.
     * :math:`\nu` is a smoothness parameter (takes values 1/2, 3/2, or 5/2). Smaller values are less smooth.
diff --git a/gpytorch/settings.py b/gpytorch/settings.py
--- a/gpytorch/settings.py
+++ b/gpytorch/settings.py
@@ -471,7 +471,7 @@ class max_preconditioner_size(_value_context):
     """
     The maximum size of preconditioner to use. 0 corresponds to turning
     preconditioning off. When enabled, usually a value of around ~10 works fairly well.
-    Default: 0
+    Default: 15
     """
 
     _global_value = 15


Test patch:
Repository: cornellius-gp/gpytorch

Issue:
ZeroMean for Batch Independent Multioutput GP
I'm following the Batch Independent Multioutput GP example, but instead of using a constant mean, I would like each dimension to use a zero mean (maybe this is a bad idea?).

```
class ZeroMeanIndependentMultitaskGPModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood, nx):
        super().__init__(train_x, train_y, likelihood)
        self.n = nx     #output dimension
        #self.mean_module = gpytorch.means.MultitaskMean([gpytorch.means.ZeroMean()]*self.n,
        #                                                num_tasks=self.n)
        #self.mean_module = gpytorch.means.ConstantMean(batch_shape=torch.Size([self.n]))
        self.mean_module = gpytorch.means.ZeroMean()
        self.covar_module = gpytorch.kernels.ScaleKernel(
            gpytorch.kernels.RBFKernel(batch_shape=torch.Size([self.n])),
            batch_shape=torch.Size([self.n])
        )

    def forward(self, x):
        mean_x = self.mean_module(x) # is this needed for ZeroMean?
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultitaskMultivariateNormal.from_batch_mvn(
            gpytorch.distributions.MultivariateNormal(mean_x, covar_x)
        )
```


When training with this, I get the error `RuntimeError: mean should be a matrix or a batch matrix (batch mode)`. It works as intended with constant mean. As you can see, I've tried a couple different things, but they don't seem to work either. I can't seem to find other people with the same issue online. Is it possible to do this with ZeroMean?



Discussion:

Resolving patch:
diff --git a/gpytorch/means/zero_mean.py b/gpytorch/means/zero_mean.py
--- a/gpytorch/means/zero_mean.py
+++ b/gpytorch/means/zero_mean.py
@@ -2,9 +2,18 @@
 
 import torch
 
+from ..utils.broadcasting import _mul_broadcast_shape
 from .mean import Mean
 
 
 class ZeroMean(Mean):
+    def __init__(self, batch_shape=torch.Size(), **kwargs):
+        super(ZeroMean, self).__init__()
+        self.batch_shape = batch_shape
+
     def forward(self, input):
-        return torch.zeros(input.shape[:-1], dtype=input.dtype, device=input.device)
+        mean = torch.zeros(*self.batch_shape, 1, dtype=input.dtype, device=input.device)
+        if input.shape[:-2] == self.batch_shape:
+            return mean.expand(input.shape[:-1])
+        else:
+            return mean.expand(_mul_broadcast_shape(input.shape[:-1], mean.shape))


Test patch:
Repository: cornellius-gp/gpytorch

Issue:
[Bug] inplace operation in RBFKernelGrad when diag=True
# 🐛 Bug

RBFKernelGrad uses an inplace operation when diag=True that breaks backward.

## To reproduce

```python
import torch
from gpytorch.kernels import RBFKernelGrad

# This works
k = RBFKernelGrad()
train_x=torch.tensor([1., 2., 3.])
z = k(train_x, train_x)
z[0, 0].backward()
print(k.raw_lengthscale.grad)

# This fails
z = k(train_x, train_x, diag=True)
z[0].backward()
print(k.raw_lengthscale.grad)
```

** Stack trace/error message **
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-17-ecb6525006fe> in <module>
      1 z = k(train_x, train_x, diag=True)
----> 2 z[0].backward()
      3 print(k.raw_lengthscale.grad)

/mnt/xarfuse/uid-66331/be3771ae-seed-nspid4026531836-ns-4026531840/torch/tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    231                 create_graph=create_graph,
    232                 inputs=inputs)
--> 233         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    234 
    235     def register_hook(self, hook):

/mnt/xarfuse/uid-66331/be3771ae-seed-nspid4026531836-ns-4026531840/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    144     Variable._execution_engine.run_backward(
    145         tensors, grad_tensors_, retain_graph, create_graph, inputs,
--> 146         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
    147 
    148 

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 1]], which is output 0 of SoftplusBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
```

## Expected Behavior

Same as what happens without `diag=True`.

## System information

**Please complete the following information:**
- GPyTorch Version 1.3.0
- PyTorch Version 1.7.1
- Linux


Discussion:

Resolving patch:
diff --git a/gpytorch/kernels/rbf_kernel_grad.py b/gpytorch/kernels/rbf_kernel_grad.py
--- a/gpytorch/kernels/rbf_kernel_grad.py
+++ b/gpytorch/kernels/rbf_kernel_grad.py
@@ -108,7 +108,7 @@ def forward(self, x1, x2, diag=False, **params):
                 raise RuntimeError("diag=True only works when x1 == x2")
 
             kernel_diag = super(RBFKernelGrad, self).forward(x1, x2, diag=True)
-            grad_diag = torch.ones(*batch_shape, n2, d, device=x1.device, dtype=x1.dtype) / self.lengthscale.pow_(2)
+            grad_diag = torch.ones(*batch_shape, n2, d, device=x1.device, dtype=x1.dtype) / self.lengthscale.pow(2)
             grad_diag = grad_diag.transpose(-1, -2).reshape(*batch_shape, n2 * d)
             k_diag = torch.cat((kernel_diag, grad_diag), dim=-1)
             pi = torch.arange(n2 * (d + 1)).view(d + 1, n2).t().reshape((n2 * (d + 1)))


Test patch:
Repository: cornellius-gp/gpytorch

Issue:
Code in variational_strategy.py
# 🐛 Bug

Dear Gpytorch Developers

Here is a possible issue with variational_strategy.py

I am wondering whether it is a special consideration or something missing.  According to the formula in Line 121, it seems an L is missing in front of self.prior_distribution.mean in Line 115.   I understand variable inducing_values has already included L.inv() according to whitening of the inducing posterior, but this is not the case for the prior.   It is fortunate that self.prior_distribution.mean is always 0, so no errors.

Thank you for the great package

J.

## To reproduce

** Code snippet to reproduce **
```python
# Your code goes here
# Please make sure it does not require any external dependencies (other than PyTorch!)
# (We much prefer small snippets rather than links to existing libraries!)
```

** Stack trace/error message **
```
// Paste the bad output here!
```

## Expected Behavior

<!-- A clear and concise description of what you expected to happen. -->

## System information

**Please complete the following information:**
- <!-- GPyTorch Version (run `print(gpytorch.__version__)` -->
- <!-- PyTorch Version (run `print(torch.__version__)` -->
- <!-- Computer OS -->

## Additional context
Add any other context about the problem here.



Discussion:

Resolving patch:
diff --git a/gpytorch/variational/variational_strategy.py b/gpytorch/variational/variational_strategy.py
--- a/gpytorch/variational/variational_strategy.py
+++ b/gpytorch/variational/variational_strategy.py
@@ -114,12 +114,7 @@ def forward(self, x, inducing_points, inducing_values, variational_inducing_cova
 
         # Compute the mean of q(f)
         # k_XZ K_ZZ^{-1/2} (m - K_ZZ^{-1/2} \mu_Z) + \mu_X
-        predictive_mean = (
-            torch.matmul(
-                interp_term.transpose(-1, -2), (inducing_values - self.prior_distribution.mean).unsqueeze(-1)
-            ).squeeze(-1)
-            + test_mean
-        )
+        predictive_mean = (interp_term.transpose(-1, -2) @ inducing_values.unsqueeze(-1)).squeeze(-1) + test_mean
 
         # Compute the covariance of q(f)
         # K_XX + k_XZ K_ZZ^{-1/2} (S - I) K_ZZ^{-1/2} k_ZX


Test patch:
Repository: cornellius-gp/gpytorch

Issue:
[Bug] CG convergence warning incorrectly triggered when max_iter <= 10
# 🐛 Bug

The linear conjugate gradient method can incorrectly flag a numerical error higher than the tolerance when `max_iter <= 10`.

## To reproduce

** Code snippet to reproduce **
```python
from gpytorch.utils import linear_cg
import torch

mat = torch.rand((2,2))
matmul_closure = lambda x: torch.matmul(mat, x)
rhs = torch.rand((2,1))

solve = linear_cg(matmul_closure, rhs, max_iter=2, max_tridiag_iter=2)
```

** Stack trace/error message **
```
NumericalWarning: CG terminated in 2 iterations with average residual norm 0.48401400446891785 which is larger than the tolerance of 1 specified by gpytorch.settings.cg_tolerance. If performance is affected, consider raising the maximum number of CG iterations by running code in a gpytorch.settings.max_cg_iterations(value) context.
```

## Expected Behavior

We would expect this warning to not be triggered, as the average residual norm is smaller than the tolerance of 1.

## System information

**Please complete the following information:**
- GPyTorch Version 1.4.0
- PyTorch Version 1.7.1
- Computer OS: Ubuntu 18.04

## Additional context
The problem here is that `tolerance_reached=True` is never triggered on Line 286 of `linear_cg.py` since `k < 10` whenever `max_iter <= 10`. This then incorrectly triggers the warning on Line 317.


Discussion:

Resolving patch:
diff --git a/gpytorch/utils/linear_cg.py b/gpytorch/utils/linear_cg.py
--- a/gpytorch/utils/linear_cg.py
+++ b/gpytorch/utils/linear_cg.py
@@ -282,7 +282,11 @@ def linear_cg(
         residual_norm.masked_fill_(rhs_is_zero, 0)
         torch.lt(residual_norm, stop_updating_after, out=has_converged)
 
-        if k >= 10 and bool(residual_norm.mean() < tolerance) and not (n_tridiag and k < n_tridiag_iter):
+        if (
+            k >= min(10, max_iter - 1)
+            and bool(residual_norm.mean() < tolerance)
+            and not (n_tridiag and k < min(n_tridiag_iter, max_iter - 1))
+        ):
             tolerance_reached = True
             break
 


Test patch: