Repository: cornellius-gp/gpytorch

Issue:
[Bug] Spectral Mixture Kernel initialization bug
# 🐛 Bug

Spectral kernel initialize_from_data fails for particular types of data. In this case, if one of the dimensions of the training data is equal for all points, the initialization fails.

## To reproduce

** Code snippet to reproduce **
```python

from gpytorch.kernels import SpectralMixtureKernel

X = torch.randint(0, 2, (100, 200))
Y = torch.randn(100)

for i in X:
    i[0] = 0
    
covar_module = SpectralMixtureKernel(num_mixtures=4, ard_num_dims=200)
covar_module.initialize_from_data(X, Y)

```
```
IndexError                                Traceback (most recent call last)
<ipython-input-175-3388eb419418> in <module>
      8 
      9 covar_module = SpectralMixtureKernel(num_mixtures=4, ard_num_dims=200)
---> 10 covar_module.initialize_from_data(X, Y)

~/anaconda3/envs/gpmodel/lib/python3.7/site-packages/gpytorch/kernels/spectral_mixture_kernel.py in initialize_from_data(self, train_x, train_y, **kwargs)
    162         min_dist = torch.zeros(1, self.ard_num_dims, dtype=train_x.dtype, device=train_x.device)
    163         for ind in range(self.ard_num_dims):
--> 164             min_dist[:, ind] = min_dist_sort[(torch.nonzero(min_dist_sort[:, ind]))[0], ind]
    165 
    166         # Inverse of lengthscales should be drawn from truncated Gaussian | N(0, max_dist^2) |

IndexError: index 0 is out of bounds for dimension 0 with size 0

```

## Expected Behavior

The kernel should be initialized for the data.

## System information

**Please complete the following information:**

- <!-- GPyTorch Version (run `print(gpytorch.__version__)` --> 1.0.1
- <!-- PyTorch Version (run `print(torch.__version__)` --> 1.4.0
- <!-- Computer OS --> OSX

## Additional context
Add any other context about the problem here.



Discussion:
The initialization strategy in the kernel doesn't really work if the minimum distance between data points in any dimension is 0. I'm opening up a PR that includes an alternative (arguably better) initialization scheme based on the empirical spectrum that does work in this case.

Resolving patch:
diff --git a/gpytorch/kernels/spectral_mixture_kernel.py b/gpytorch/kernels/spectral_mixture_kernel.py
--- a/gpytorch/kernels/spectral_mixture_kernel.py
+++ b/gpytorch/kernels/spectral_mixture_kernel.py
@@ -148,6 +148,48 @@ def _set_mixture_weights(self, value):
             value = torch.as_tensor(value).to(self.raw_mixture_weights)
         self.initialize(raw_mixture_weights=self.raw_mixture_weights_constraint.inverse_transform(value))
 
+    def initialize_from_data_empspect(self, train_x, train_y):
+        """
+        Initialize mixture components based on the empirical spectrum of the data.
+
+        This will often be better than the standard initialize_from_data method.
+        """
+        import numpy as np
+        from scipy.fftpack import fft
+        from scipy.integrate import cumtrapz
+
+        N = train_x.size(-2)
+        emp_spect = np.abs(fft(train_y.cpu().detach().numpy())) ** 2 / N
+        M = math.floor(N / 2)
+
+        freq1 = np.arange(M + 1)
+        freq2 = np.arange(-M + 1, 0)
+        freq = np.hstack((freq1, freq2)) / N
+        freq = freq[: M + 1]
+        emp_spect = emp_spect[: M + 1]
+
+        total_area = np.trapz(emp_spect, freq)
+        spec_cdf = np.hstack((np.zeros(1), cumtrapz(emp_spect, freq)))
+        spec_cdf = spec_cdf / total_area
+
+        a = np.random.rand(1000, self.ard_num_dims)
+        p, q = np.histogram(a, spec_cdf)
+        bins = np.digitize(a, q)
+        slopes = (spec_cdf[bins] - spec_cdf[bins - 1]) / (freq[bins] - freq[bins - 1])
+        intercepts = spec_cdf[bins - 1] - slopes * freq[bins - 1]
+        inv_spec = (a - intercepts) / slopes
+
+        from sklearn.mixture import GaussianMixture
+
+        GMM = GaussianMixture(n_components=self.num_mixtures, covariance_type="diag").fit(inv_spec)
+        means = GMM.means_
+        varz = GMM.covariances_
+        weights = GMM.weights_
+
+        self.mixture_means = means
+        self.mixture_scales = varz
+        self.mixture_weights = weights
+
     def initialize_from_data(self, train_x, train_y, **kwargs):
         if not torch.is_tensor(train_x) or not torch.is_tensor(train_y):
             raise RuntimeError("train_x and train_y should be tensors")


Test patch: