anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: ktbyers/netmiko
Issue: CIiena Base Prompt not correctly setting base prompt

ciena_saos.py line 31 in "set_base_prompt" method has a typo:

The following is checking if the pattern matches the prompt.  If it does it's raising an error. It should raise an error if they don't match. 
        if re.search(pattern, prompt):
            raise ValueError(f"Router prompt not found: {repr(prompt)}")

Should be:
        if not re.search(pattern, prompt):
            raise ValueError(f"Router prompt not found: {repr(prompt)}")

Changed files:
netmiko/ciena/ciena_saos.py

diff --git a/netmiko/ciena/ciena_saos.py b/netmiko/ciena/ciena_saos.py
index f3c0c21e..b74a61c6 100644
--- a/netmiko/ciena/ciena_saos.py
+++ b/netmiko/ciena/ciena_saos.py
@@ -28,7 +28,7 @@ class CienaSaosBase(NoEnable, NoConfig, BaseConnection):
         prompt = self.find_prompt(delay_factor=delay_factor)
 
         pattern = rf"^.+{self.prompt_pattern}$"
-        if re.search(pattern, prompt):
+        if not re.search(pattern, prompt):
             raise ValueError(f"Router prompt not found: {repr(prompt)}")
         # Strip off trailing terminator
         self.base_prompt = prompt[:-1]
Repository: ktbyers/netmiko
Issue: Ubiquity Edge Switch is failing on save_config due to the wrong search pattern.

In the method save_config of the class `UbiquitiEdgeSSH` are you waiting for the wrong search pattern.
At the moment you are expecting `\\(UBNT\\ EdgeSwitch\\)\\ \\#`.

After the command `write memory` the cli print following out:
```
This operation may take a few minutes.
Management interfaces will not be available during this time.

Are you sure you want to save? (y/n) 
```

It's expecting an additional input.

This is how i handle it, maybe you can do it the same way:
```
    def save_config(self):
        try:
            output = self.send_command(command_string="write memory", expect_string=r"(y/n)", strip_command=False, strip_prompt=False)
            output += self.send_command(command_string="y", expect_string=r"#", strip_command=False, strip_prompt=False)
            return output
        except Exception as e:
            raise e
```

Changed files:
netmiko/ubiquiti/edge_ssh.py

diff --git a/netmiko/ubiquiti/edge_ssh.py b/netmiko/ubiquiti/edge_ssh.py
index bff0473c..7ec08c41 100644
--- a/netmiko/ubiquiti/edge_ssh.py
+++ b/netmiko/ubiquiti/edge_ssh.py
@@ -46,10 +46,38 @@ class UbiquitiEdgeSSH(CiscoSSHConnection):
     def save_config(
         self,
         cmd: str = "write memory",
-        confirm: bool = False,
-        confirm_response: str = "",
+        confirm: bool = True,
+        confirm_response: str = "y",
     ) -> str:
-        """Saves configuration."""
-        return super().save_config(
-            cmd=cmd, confirm=confirm, confirm_response=confirm_response
-        )
+        """
+        EdgeSwitch might do the following:
+
+        Are you sure you want to save? (y/n) y
+
+        It is not clear to me that it always does this (or sometimes doesn't).
+
+        Consequently, save_config() handles both cases.
+        """
+        self.enable()
+        confirm_msg = "Are you sure"
+        pattern = rf"({confirm_msg}|\\#)"
+        if confirm:
+            output = self._send_command_str(
+                command_string=cmd,
+                expect_string=pattern,
+                strip_prompt=False,
+                strip_command=False,
+            )
+            if confirm_response and confirm_msg in output:
+                output += self._send_command_str(
+                    confirm_response,
+                    expect_string=r"\\#",
+                    strip_prompt=False,
+                    strip_command=False,
+                )
+        else:
+            # Shouldn't really be here--but handle base case if someone disables "confirm".
+            output = self._send_command_str(
+                command_string=cmd, strip_prompt=False, strip_command=False
+            )
+        return output
Repository: cornellius-gp/gpytorch
Issue: [Bug] Cat_rows evaluation fails to capture correct device

# 🐛 Bug

Evaluating `LazyTensor.cat_rows` on the gpu returns a cpu tensor.

## To reproduce

** Code snippet to reproduce **
```python
from gpytorch.kernels import MaternKernel
import torch

x = torch.randn(10, 3).cuda()
y = torch.randn(5, 3).cuda()

kernel = MaternKernel().cuda()

kxx = kernel(x)
kxy = kernel(y, x)
kyy = kernel(y)

joint_covariance = kxx.cat_rows(kxy, kyy)
joint_covariance.matmul(torch.randn(15, 3).cuda()).device # on the gpu properly
joint_covariance.evaluate().device # returns cpu
```

** Stack trace/error message **
See above. It's a missing device_id argument in the CatLazyTensor calls.

## Expected Behavior

returns on the gpu. 

## System information

**Please complete the following information:**
- gpytorch master

## Additional context
I'll put up a PR shortly. I think that this is causing the test failures in https://github.com/pytorch/botorch/pull/883


Changed files:
gpytorch/lazy/lazy_tensor.py

diff --git a/gpytorch/lazy/lazy_tensor.py b/gpytorch/lazy/lazy_tensor.py
index 725ba4c3..cacf7a0e 100644
--- a/gpytorch/lazy/lazy_tensor.py
+++ b/gpytorch/lazy/lazy_tensor.py
@@ -780,9 +780,9 @@ class LazyTensor(ABC):
             A = self
 
         # form matrix C = [A B; B^T D], where A = self, B = cross_mat, D = new_mat
-        upper_row = CatLazyTensor(A, B, dim=-2)
-        lower_row = CatLazyTensor(B.transpose(-1, -2), D, dim=-2)
-        new_lazy_tensor = CatLazyTensor(upper_row, lower_row, dim=-1)
+        upper_row = CatLazyTensor(A, B, dim=-2, output_device=A.device)
+        lower_row = CatLazyTensor(B.transpose(-1, -2), D, dim=-2, output_device=A.device)
+        new_lazy_tensor = CatLazyTensor(upper_row, lower_row, dim=-1, output_device=A.device)
 
         # if the old lazy tensor does not have either a root decomposition or a root inverse decomposition
         # don't create one
Repository: cornellius-gp/gpytorch
Issue: [Bug] Incorrect gradients for kernels using torch.cdist

# 🐛 Bug
Auto-differientiating torch.cdist on the GPU is broken (https://github.com/pytorch/pytorch/issues/22353). This means kernels using torch.cdist have incorrect gradients on the GPU

## To reproduce
[cdist_issue_gpytorch.ipynb.txt](https://github.com/cornellius-gp/gpytorch/files/3340948/cdist_issue_gpytorch.ipynb.txt)

## Expected Behavior

Numerical and analytic gradients match

## System information

**Please complete the following information:**
GPyTorch Version: master
PyTorch Version (e.g., 1.0): master
OS (e.g., Linux): CentOS Linux 7 (Core)
CUDA/cuDNN version:CUDA Version 9.2.88
Tesla M40


Changed files:
gpytorch/kernels/kernel.py

diff --git a/gpytorch/kernels/kernel.py b/gpytorch/kernels/kernel.py
index dce3c238..9e33f874 100644
--- a/gpytorch/kernels/kernel.py
+++ b/gpytorch/kernels/kernel.py
@@ -22,7 +22,14 @@ class Distance(torch.nn.Module):
         self._postprocess = postprocess_script
 
     def _sq_dist(self, x1, x2, postprocess, x1_eq_x2=False):
-        if hasattr(torch, 'cdist') and x1.dim() < 3 and x1.size(-2) <= 512 and x2.size(-2) <= 512:
+        if (
+            hasattr(torch, 'cdist')
+            and x1.dim() < 3
+            and x1.size(-2) <= 512
+            and x2.size(-2) <= 512
+            # TODO: remove when https://github.com/pytorch/pytorch/issues/22353 is fixed
+            and not (x1.is_cuda or x2.is_cuda)
+        ):
             # torch cdist doesn't support batch, also only available after pytorch 1.1
             # cdist is a bit faster on small tensors. Check again after pytorch PR #20605
             res = torch.cdist(x1, x2).pow(2)
@@ -47,7 +54,14 @@ class Distance(torch.nn.Module):
         return self._postprocess(res) if postprocess else res
 
     def _dist(self, x1, x2, postprocess, x1_eq_x2=False):
-        if hasattr(torch, 'cdist') and x1.dim() < 3 and x1.size(-2) <= 512 and x2.size(-2) <= 512:
+        if (
+            hasattr(torch, 'cdist')
+            and x1.dim() < 3
+            and x1.size(-2) <= 512
+            and x2.size(-2) <= 512
+            # TODO: remove when https://github.com/pytorch/pytorch/issues/22353 is fixed
+            and not (x1.is_cuda or x2.is_cuda)
+        ):
             # torch cdist doesn't support batch, also only available after pytorch 1.1
             # cdist is a bit faster on small tensors. Check again after pytorch PR #20605
             res = torch.cdist(x1, x2)
Repository: cornellius-gp/gpytorch
Issue: [Bug] Interval constraint miscalculates bounds

# 🐛 Bug

The transform function in the Interval constraint class allows tensors to be above the upper bound.


## To reproduce

** Code snippet to reproduce **

import torch
import gpytorch

lower_bound = 0.2
upper_bound = 0.3
tensor = torch.Tensor([lower_bound + upper_bound + 0.1])

constrained_tensor = gpytorch.constraints.Interval(lower_bound = lower_bound, upper_bound=upper_bound).transform(tensor)
assert constrained_tensor >= lower_bound, 'constrained tensor smaller than lower bound'
assert constrained_tensor <= upper_bound, 'constrained tensor larger than upper bound'

** Stack trace/error message **

AssertionError: constrained tensor larger than upper bound

## Expected Behavior

We would want the constrained tensor to be between lower bound and upper bound

## System information

**Please complete the following information:**
- <!-- GPyTorch Version (run `print(gpytorch.__version__)` --> 0.3.2
- <!-- PyTorch Version (run `print(torch.__version__)` --> 1.1.0
- <!-- Computer OS --> macOS


Changed files:
test/constraints/test_constraints.py, gpytorch/constraints/constraints.py

diff --git a/gpytorch/constraints/constraints.py b/gpytorch/constraints/constraints.py
index 30b5b4af..6853f15e 100644
--- a/gpytorch/constraints/constraints.py
+++ b/gpytorch/constraints/constraints.py
@@ -94,7 +94,7 @@ class Interval(Module):
                     "GreaterThan or LessThan instead."
                 )
 
-        transformed_tensor = (self._transform(tensor) * self.upper_bound) + self.lower_bound
+        transformed_tensor = (self._transform(tensor) * (self.upper_bound - self.lower_bound)) + self.lower_bound
 
         return transformed_tensor
 
@@ -115,7 +115,7 @@ class Interval(Module):
                     "GreaterThan or LessThan instead."
                 )
 
-        tensor = self._inv_transform((transformed_tensor - self.lower_bound) / self.upper_bound)
+        tensor = self._inv_transform((transformed_tensor - self.lower_bound) / (self.upper_bound - self.lower_bound))
 
         return tensor
 
diff --git a/test/constraints/test_constraints.py b/test/constraints/test_constraints.py
index 611c0ac4..1d47a5f5 100644
--- a/test/constraints/test_constraints.py
+++ b/test/constraints/test_constraints.py
@@ -28,7 +28,7 @@ class TestInterval(unittest.TestCase, BaseTestCase):
         v = torch.tensor(-3.0)
 
         value = constraint.transform(v)
-        actual_value = (5.0 * sigmoid(v)) + 1.0
+        actual_value = ((5.0 - 1.0) * sigmoid(v)) + 1.0
 
         self.assertAllClose(value, actual_value)
 
@@ -48,8 +48,8 @@ class TestInterval(unittest.TestCase, BaseTestCase):
 
         value = constraint.transform(v)
         actual_value = v.clone()
-        actual_value[0] = 3.0 * sigmoid(v[0]) + 1.0
-        actual_value[1] = 4.0 * sigmoid(v[1]) + 2.0
+        actual_value[0] = (3.0 - 1.0) * sigmoid(v[0]) + 1.0
+        actual_value[1] = (4.0 - 2.0) * sigmoid(v[1]) + 2.0
 
         self.assertAllClose(value, actual_value)
Repository: cornellius-gp/gpytorch
Issue: [Bug] TypeError when calling `backward` on `gpytorch.functions.logdet` 

# 🐛 Bug

I think there's a missing `list()` in https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/functions/_inv_quad_log_det.py#L221. I'm not super familiar with gpytorch internals so hopefully this is correct -- if so, happy to contribute the one-liner fix. 

## To reproduce

** Code snippet to reproduce **
```python
### works (I'm guessing something dispatches elsewhere for small matrices?)
import torch
from torch.autograd import backward
import gpytorch
from gpytorch.functions import logdet, inv_matmul
n = 100
inp = torch.arange(n, dtype=torch.float)
kern = gpytorch.kernels.RBFKernel()(inp)
ld = logdet(kern)
backward(ld)

### doesn't work
import torch
from torch.autograd import backward
import gpytorch
from gpytorch.functions import logdet, inv_matmul
n = 1000
inp = torch.arange(n, dtype=torch.float)
kern = gpytorch.kernels.RBFKernel()(inp)
ld = logdet(kern)
backward(ld)
```

** Stack trace/error message **
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-46-593fbced29ac> in <module>()
      3 kern = gpytorch.kernels.RBFKernel()(inp)
      4 ld = logdet(kern)
----> 5 backward(ld)

<PATH SNIPPED>/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     91     Variable._execution_engine.run_backward(
     92         tensors, grad_tensors, retain_graph, create_graph,
---> 93         allow_unreachable=True)  # allow_unreachable flag
     94 
     95 

<PATH SNIPPED>/lib/python3.7/site-packages/torch/autograd/function.py in apply(self, *args)
     75 
     76     def apply(self, *args):
---> 77         return self._forward_cls.backward(self, *args)
     78 
     79 

<PATH SNIPPED>lib/python3.7/site-packages/gpytorch/functions/_inv_quad_log_det.py in backward(ctx, inv_quad_grad_output, logdet_grad_output)
    221             res = matrix_arg_grads
    222 
--> 223         return tuple([None] * 9 + res)

TypeError: can only concatenate list (not "tuple") to list
```

## Expected Behavior

No error. 

## System information

**Please complete the following information:**
- GPyTorch version: 0.3.2 <!-- GPyTorch Version (run `print(gpytorch.__version__)` -->
- pytorch version 1.1.0. 
- Mac OSX. 



Changed files:
gpytorch/functions/_inv_quad_log_det.py

diff --git a/gpytorch/functions/_inv_quad_log_det.py b/gpytorch/functions/_inv_quad_log_det.py
index 850986e1..0629f5b9 100644
--- a/gpytorch/functions/_inv_quad_log_det.py
+++ b/gpytorch/functions/_inv_quad_log_det.py
@@ -218,6 +218,6 @@ class InvQuadLogDet(Function):
         if ctx.inv_quad:
             res = [inv_quad_rhs_grad] + list(matrix_arg_grads)
         else:
-            res = matrix_arg_grads
+            res = list(matrix_arg_grads)
 
         return tuple([None] * 9 + res)
Repository: cornellius-gp/gpytorch
Issue: Where is `fast_pred_var` moving?

I received the following warning when using `fast_pred_var`:

```
/cluster/nhunt/anaconda/envs/bayes_opt/lib/python3.7/site-packages/gpytorch/beta_features.py:17:
DeprecationWarning: `gpytorch.settings.fast_pred_var` has moved to `gpytorch.settings.fast_pred_var`.
```

It seems that I'm being warned that `fast_pred_var` has moved to its current location. Was there a typo in the warning about how we should be using this setting now?

```bash
$ pip list | grep gpytorch
gpytorch           0.2.1
```

Changed files:
gpytorch/beta_features.py, gpytorch/__init__.py

diff --git a/gpytorch/__init__.py b/gpytorch/__init__.py
index 672cc593..2360ff81 100644
--- a/gpytorch/__init__.py
+++ b/gpytorch/__init__.py
@@ -37,9 +37,6 @@ from .lazy import lazify, delazify
 
 __version__ = "0.2.1"
 
-# Old deprecated stuff
-fast_pred_var = beta_features._moved_beta_feature(settings.fast_pred_var, "gpytorch.settings.fast_pred_var")
-
 __all__ = [
     # Submodules
     "distributions",
@@ -77,7 +74,6 @@ __all__ = [
     # Other
     "__version__",
     # Deprecated
-    "fast_pred_var",
     "inv_quad_log_det",
     "log_det",
 ]
diff --git a/gpytorch/beta_features.py b/gpytorch/beta_features.py
index 714bea22..91bf4dbe 100644
--- a/gpytorch/beta_features.py
+++ b/gpytorch/beta_features.py
@@ -2,8 +2,6 @@
 
 import warnings
 from .settings import _feature_flag, _value_context
-from .settings import fast_pred_var as _fast_pred_var
-from .settings import fast_pred_samples as _fast_pred_samples
 
 
 class _moved_beta_feature(object):
@@ -22,10 +20,6 @@ class _moved_beta_feature(object):
         return getattr(self.new_cls, name)
 
 
-fast_pred_var = _moved_beta_feature(_fast_pred_var)
-fast_pred_samples = _moved_beta_feature(_fast_pred_samples)
-
-
 class checkpoint_kernel(_value_context):
     """
     Should the kernel be computed in chunks with checkpointing or not? (Default, no)
@@ -61,4 +55,4 @@ class default_preconditioner(_feature_flag):
     pass
 
 
-__all__ = ["fast_pred_var", "fast_pred_samples", "diagonal_correction", "default_preconditioner"]
+__all__ = ["checkpoint_kernel", "diagonal_correction", "default_preconditioner"]
Repository: cornellius-gp/gpytorch
Issue: Code in variational_strategy.py

# 🐛 Bug

Dear Gpytorch Developers

Here is a possible issue with variational_strategy.py

I am wondering whether it is a special consideration or something missing.  According to the formula in Line 121, it seems an L is missing in front of self.prior_distribution.mean in Line 115.   I understand variable inducing_values has already included L.inv() according to whitening of the inducing posterior, but this is not the case for the prior.   It is fortunate that self.prior_distribution.mean is always 0, so no errors.

Thank you for the great package

J.

## To reproduce

** Code snippet to reproduce **
```python
# Your code goes here
# Please make sure it does not require any external dependencies (other than PyTorch!)
# (We much prefer small snippets rather than links to existing libraries!)
```

** Stack trace/error message **
```
// Paste the bad output here!
```

## Expected Behavior

<!-- A clear and concise description of what you expected to happen. -->

## System information

**Please complete the following information:**
- <!-- GPyTorch Version (run `print(gpytorch.__version__)` -->
- <!-- PyTorch Version (run `print(torch.__version__)` -->
- <!-- Computer OS -->

## Additional context
Add any other context about the problem here.


Changed files:
gpytorch/variational/variational_strategy.py

diff --git a/gpytorch/variational/variational_strategy.py b/gpytorch/variational/variational_strategy.py
index 8996e6c9..3733f938 100644
--- a/gpytorch/variational/variational_strategy.py
+++ b/gpytorch/variational/variational_strategy.py
@@ -114,12 +114,7 @@ class VariationalStrategy(_VariationalStrategy):
 
         # Compute the mean of q(f)
         # k_XZ K_ZZ^{-1/2} (m - K_ZZ^{-1/2} \\mu_Z) + \\mu_X
-        predictive_mean = (
-            torch.matmul(
-                interp_term.transpose(-1, -2), (inducing_values - self.prior_distribution.mean).unsqueeze(-1)
-            ).squeeze(-1)
-            + test_mean
-        )
+        predictive_mean = (interp_term.transpose(-1, -2) @ inducing_values.unsqueeze(-1)).squeeze(-1) + test_mean
 
         # Compute the covariance of q(f)
         # K_XX + k_XZ K_ZZ^{-1/2} (S - I) K_ZZ^{-1/2} k_ZX
Repository: cornellius-gp/gpytorch
Issue: [Bug] Cannot serialize/deserialize SmoothedBoxPrior when some args are broadcast

# 🐛 Bug

It seems like `SmoothedBoxPrior` for >1d doesn't work with serialization/deserialization when only some args are broadcast. 

## To reproduce

```python
import torch
import gpytorch
pr = gpytorch.priors.SmoothedBoxPrior(torch.zeros(2), torch.ones(2))
pr.load_state_dict(pr.state_dict())
```

** Stack trace/error message **
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-6b4b2e881beb> in <module>
      2 import gpytorch
      3 pr = gpytorch.priors.SmoothedBoxPrior(torch.zeros(2), torch.ones(2))
----> 4 pr.load_state_dict(pr.state_dict())

<...PATH..>/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
    877         if len(error_msgs) > 0:
    878             raise RuntimeError('Error(s) in loading state_dict for {}:\\n\\t{}'.format(
--> 879                                self.__class__.__name__, "\\n\\t".join(error_msgs)))
    880         return _IncompatibleKeys(missing_keys, unexpected_keys)
    881 

RuntimeError: Error(s) in loading state_dict for SmoothedBoxPrior:
	While copying the parameter named "sigma", whose dimensions in the model are torch.Size([2]) and whose dimensions in the checkpoint are torch.Size([2]), an exception occured : ('unsupported operation: more than one element of the written-to tensor refers to a single memory location. Please clone() the tensor before performing the operation.',).

```

Note that `SmoothedBoxPrior(a=torch.zeros(2), b=torch.ones(2), sigma=torch.ones(2)*0.01)` succeeds, as does `gpytorch.priors.GammaPrior(torch.ones(2),1)`.

## Expected Behavior

Successful load. 

## System information

**Please complete the following information:**
- gpytorch version: 1.1.1
- pytorch version: 1.5.0
- OS: tested on Centos and Mac OSX. 




Changed files:
gpytorch/priors/smoothed_box_prior.py

diff --git a/gpytorch/priors/smoothed_box_prior.py b/gpytorch/priors/smoothed_box_prior.py
index 3a2d8292..58ad3327 100644
--- a/gpytorch/priors/smoothed_box_prior.py
+++ b/gpytorch/priors/smoothed_box_prior.py
@@ -47,7 +47,7 @@ class SmoothedBoxPrior(Prior):
         del self.a, self.b, self.sigma
         self.register_buffer("a", _a)
         self.register_buffer("b", _b)
-        self.register_buffer("sigma", _sigma)
+        self.register_buffer("sigma", _sigma.clone())
         self.tails = NormalPrior(torch.zeros_like(_a), _sigma, validate_args=validate_args)
         self._transform = transform
Repository: cornellius-gp/gpytorch
Issue: [Bug] Typo in doc - Matern Kernel

# 🐛 Bug

Should the documentation here https://docs.gpytorch.ai/en/v1.1.1/kernels.html#maternkernel have $ \\Theta^{-2}$ in the definition of `d` like the other kernels (example RBF)? 


Changed files:
gpytorch/settings.py, gpytorch/kernels/matern_kernel.py

diff --git a/gpytorch/kernels/matern_kernel.py b/gpytorch/kernels/matern_kernel.py
index f8b2793f..4701b092 100644
--- a/gpytorch/kernels/matern_kernel.py
+++ b/gpytorch/kernels/matern_kernel.py
@@ -23,7 +23,7 @@ class MaternKernel(Kernel):
 
     where
 
-    * :math:`d = (\\mathbf{x_1} - \\mathbf{x_2})^\\top \\Theta^{-1} (\\mathbf{x_1} - \\mathbf{x_2})`
+    * :math:`d = (\\mathbf{x_1} - \\mathbf{x_2})^\\top \\Theta^{-2} (\\mathbf{x_1} - \\mathbf{x_2})`
       is the distance between
       :math:`x_1` and :math:`x_2` scaled by the :attr:`lengthscale` parameter :math:`\\Theta`.
     * :math:`\\nu` is a smoothness parameter (takes values 1/2, 3/2, or 5/2). Smaller values are less smooth.
diff --git a/gpytorch/settings.py b/gpytorch/settings.py
index cb720054..81359462 100644
--- a/gpytorch/settings.py
+++ b/gpytorch/settings.py
@@ -471,7 +471,7 @@ class max_preconditioner_size(_value_context):
     """
     The maximum size of preconditioner to use. 0 corresponds to turning
     preconditioning off. When enabled, usually a value of around ~10 works fairly well.
-    Default: 0
+    Default: 15
     """
 
     _global_value = 15
Repository: cornellius-gp/gpytorch
Issue: Repeated posterior evaluation (and backward) with same base_samples raises second backward issue

I am encountering an second backward error when doing repeated posterior evaluations with the same set of base_samples. If I use `detach_test_caches(True)` and `fast_pred_var()`, there is no issue. However, when I do not use `fast_pred_var()` and/or set `detach_test_caches(False)`, then the second backward issue is raised.

Repro:
[gpytorch_base_samples_caching_backward_issue.ipynb.txt](https://github.com/cornellius-gp/gpytorch/files/2937770/gpytorch_base_samples_caching_backward_issue.ipynb.txt)


Changed files:
gpytorch/lazy/lazy_tensor.py, gpytorch/models/exact_prediction_strategies.py

diff --git a/gpytorch/lazy/lazy_tensor.py b/gpytorch/lazy/lazy_tensor.py
index f9aed6b0..9a0eaa30 100644
--- a/gpytorch/lazy/lazy_tensor.py
+++ b/gpytorch/lazy/lazy_tensor.py
@@ -995,7 +995,11 @@ class LazyTensor(object):
 
     @property
     def requires_grad(self):
-        return any(arg.requires_grad for arg in tuple(self._args) + tuple(self._kwargs.values()))
+        return any(
+            arg.requires_grad
+            for arg in tuple(self._args) + tuple(self._kwargs.values())
+            if hasattr(arg, "requires_grad")
+        )
 
     @requires_grad.setter
     def requires_grad(self, val):
diff --git a/gpytorch/models/exact_prediction_strategies.py b/gpytorch/models/exact_prediction_strategies.py
index 6366073c..2eb61e96 100644
--- a/gpytorch/models/exact_prediction_strategies.py
+++ b/gpytorch/models/exact_prediction_strategies.py
@@ -1,6 +1,7 @@
 #!/usr/bin/env python3
 
 import torch
+import contextlib
 
 from .. import settings
 from ..distributions import MultivariateNormal
@@ -315,9 +316,18 @@ class DefaultPredictionStrategy(object):
             return ZeroLazyTensor(*test_test_covar.size())
 
         if settings.fast_pred_var.off():
-            train_train_covar = self.likelihood(
-                MultivariateNormal(torch.zeros(1), self.train_train_covar), self.train_inputs
-            ).lazy_covariance_matrix
+            if settings.detach_test_caches.on():
+                if self.train_train_covar.requires_grad:
+                    self.train_train_covar = self.train_train_covar.detach()
+                lik_context = torch.no_grad
+            else:
+                lik_context = contextlib.suppress
+
+            with lik_context():
+                train_train_covar = self.likelihood(
+                    MultivariateNormal(torch.zeros(1), self.train_train_covar), self.train_inputs
+                ).lazy_covariance_matrix
+
             test_train_covar = test_train_covar.evaluate()
             train_test_covar = test_train_covar.transpose(-1, -2)
             covar_correction_rhs = train_train_covar.inv_matmul(train_test_covar).mul(-1)
Repository: cornellius-gp/gpytorch
Issue: RBF Kernel Change Breaks Testing Code

The change to RBFKernel in 84fccd898c45c08279fb5c109e6e234f3a47588a may break something about our prediction code. 

I am not totally sure what the problem is yet, but I isolated this as the problem with `git bisect` and have a reasonable test case where results are significantly worse with the commit in compared to after a revert commit. 

It seems like the stability issues we encountered when making this change in the past don't come up in the unit tests, but do on some real datasets.

I can try to push my test case to a branch as well, although it relies on a UCI dataset.

@Balandat @gpleiss 

Changed files:
gpytorch/kernels/rbf_kernel.py

diff --git a/gpytorch/kernels/rbf_kernel.py b/gpytorch/kernels/rbf_kernel.py
index 86db70a3..65730890 100644
--- a/gpytorch/kernels/rbf_kernel.py
+++ b/gpytorch/kernels/rbf_kernel.py
@@ -3,6 +3,7 @@ from __future__ import division
 from __future__ import print_function
 from __future__ import unicode_literals
 
+import math
 from .kernel import Kernel
 
 
@@ -12,7 +13,7 @@ class RBFKernel(Kernel):
         self,
         ard_num_dims=None,
         log_lengthscale_bounds=(-10000, 10000),
-        eps=1e-5,
+        eps=1e-6,
         active_dims=None,
     ):
         super(RBFKernel, self).__init__(
@@ -24,6 +25,6 @@ class RBFKernel(Kernel):
         self.eps = eps
 
     def forward(self, x1, x2):
-        lengthscales = self.log_lengthscale.exp() + self.eps
+        lengthscales = self.log_lengthscale.exp().mul(math.sqrt(2)).clamp(self.eps, 1e5)
         diff = (x1.unsqueeze(2) - x2.unsqueeze(1)).div_(lengthscales)
-        return diff.pow_(2).sum(-1).mul_(-0.5).exp_()
+        return diff.pow_(2).sum(-1).mul_(-1).exp_()