Repository: cornellius-gp/gpytorch

Issue:
ard_num_dims handled inconsistently across kernels
According to this comment(https://github.com/cornellius-gp/gpytorch/issues/314#issuecomment-427047286), it seems like ard_num_dims should always be equal to the dimension of the input dimension, and if it isn't, it should throw an error. However, SpectralMixtureModel is the only model that does so, the rest (RBFKernel and MaternKernel) fail in unexpected ways.




Discussion:
`ard_num_dims` is an optional feature for most kernels. If it is not set, then the kernel uses the same length scale for all dimensions. This is handled by PyTorch Tensor broadcasting. If `ard_num_dims=None`, then the lengthscale is a `1 x 1` Tensor. If `ard_num_dims=d`, then it is a `1 x d` tensor.
The spectral mixture kernel is a bit of an exception, in part because it behaves differently than most other kernels. [By design](https://arxiv.org/pdf/1302.4245.pdf), the SM kernel for a multi-dimensional dataset is defined as the product of 1D SM kernels across each dimension. Since the SM kernel is highly parametric, it would not make sense for each dimension/mixture component to share the same lengthscale parameters.

I do agree that maybe this could be documented better, but the SM kernel is a bit of an exception. If you have suggestions, let me know!
And can you give an example of how RBF/Matern kernels fail if `ard_num_dims` is not set? It should probably fail if `ard_num_dims=4` and the input data is `nx6`, for example. However, if `ard_num_dims` is not set then it should work.
Sorry for not being clear. Specifically, the bug report is that when ard_num_dims is set to a value that isn't equal to the input dimension, it sometimes fails silently (such as when ard_num_dims is set to 0).

These were essentially the things I was running that confused me: https://colab.research.google.com/drive/1V0PbDIH2KaCc8toshgshMa2-Gb0BS3bf

Specifically, I believe the current behavior for the RBF kernel is that if ard_num_dims>input_dim, it essentially works silently. If ard_num_dims == 0 and input_dim == 1, it fails silently. And if ard_num_dims == 0 and input_dim >1, it throws an error.

I was having some difficulty figuring out the intended behavior of ard_num_dims.

I think it'd be a good idea if it just failed loudly (ie: threw an error) every time.
Totally agree. I’ll work on a PR for some loud errors.

Resolving patch:
diff --git a/gpytorch/kernels/kernel.py b/gpytorch/kernels/kernel.py
--- a/gpytorch/kernels/kernel.py
+++ b/gpytorch/kernels/kernel.py
@@ -226,6 +226,7 @@ def __call__(self, x1, x2=None, diag=False, batch_dims=None, **params):
             x2_ = x1_
 
         # Check batch_dims args
+        # Check that ard_num_dims matches the supplied number of dimensions
         if settings.debug.on():
             if batch_dims is not None:
                 if not isinstance(batch_dims, tuple) or (batch_dims != (0,) and batch_dims != (0, 2)):
@@ -233,6 +234,12 @@ def __call__(self, x1, x2=None, diag=False, batch_dims=None, **params):
                         "batch_dims currently accepts either None, (0,), or (0, 2). Got {}.".format(batch_dims)
                     )
 
+            if self.ard_num_dims is not None and self.ard_num_dims != x1_.size(-1):
+                raise RuntimeError(
+                    "Expected the input to have {} dimensionality "
+                    "(based on the ard_num_dims argument). Got {}.".format(self.ard_num_dims, x1_.size(-1))
+                )
+
         if diag:
             res = super(Kernel, self).__call__(x1_, x2_, diag=True, batch_dims=batch_dims, **params)
 


Test patch:
diff --git a/test/examples/test_kissgp_additive_classification.py b/test/examples/test_kissgp_additive_classification.py
--- a/test/examples/test_kissgp_additive_classification.py
+++ b/test/examples/test_kissgp_additive_classification.py
@@ -32,7 +32,7 @@ def __init__(self):
         self.mean_module = ConstantMean(prior=SmoothedBoxPrior(-1e-5, 1e-5))
         self.covar_module = ScaleKernel(
             RBFKernel(
-                ard_num_dims=2, log_lengthscale_prior=SmoothedBoxPrior(exp(-5), exp(6), sigma=0.1, log_transform=True)
+                ard_num_dims=1, log_lengthscale_prior=SmoothedBoxPrior(exp(-5), exp(6), sigma=0.1, log_transform=True)
             ),
             log_outputscale_prior=SmoothedBoxPrior(exp(-5), exp(6), sigma=0.1, log_transform=True),
         )