Error Cholesky CPU

RR_1: <p>Hi, I’m getting this RuntimeError while running an optimization algorithm. I was able to run without problems 2 days ago, but now I occur in this error every time. The error doesn’t show up always on the same iteration, but sometimes the algorithm can run for 23 iterations or sometimes only one or less.<br>
This is the error message:</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
 in 
     44         tilts, outputs, rewards = Algorithm5.update_data(next_config)
     45         #aggiorno il modello
---&gt; 46         Algorithm5.fit_model(next_config)
     47 
     48         end_k = time.time()

 in fit_model(self, new_config)
     54             optimizer.zero_grad()
     55             output = self.model(self.model.train_inputs[0])
---&gt; 56             loss = -mll(output, self.model.train_targets)
     57             loss.backward()
     58             optimizer.step()

~\anaconda3\lib\site-packages\gpytorch\module.py in __call__(self, *inputs, **kwargs)
     26 
     27     def __call__(self, *inputs, **kwargs):
---&gt; 28         outputs = self.forward(*inputs, **kwargs)
     29         if isinstance(outputs, list):
     30             return [_validate_module_outputs(output) for output in outputs]

~\anaconda3\lib\site-packages\gpytorch\mlls\exact_marginal_log_likelihood.py in forward(self, function_dist, target, *params)
     49         # Get the log prob of the marginal distribution
     50         output = self.likelihood(function_dist, *params)
---&gt; 51         res = output.log_prob(target)
     52 
     53         # Add additional terms (SGPR / learned inducing points, heteroskedastic likelihood models)

~\anaconda3\lib\site-packages\gpytorch\distributions\multivariate_normal.py in log_prob(self, value)
    133 
    134         # Get log determininat and first part of quadratic form
--&gt; 135         inv_quad, logdet = covar.inv_quad_logdet(inv_quad_rhs=diff.unsqueeze(-1), logdet=True)
    136 
    137         res = -0.5 * sum([inv_quad, logdet, diff.size(-1) * math.log(2 * math.pi)])

~\anaconda3\lib\site-packages\gpytorch\lazy\lazy_tensor.py in inv_quad_logdet(self, inv_quad_rhs, logdet, reduce_inv_quad)
   1000             from .chol_lazy_tensor import CholLazyTensor
   1001 
-&gt; 1002             cholesky = CholLazyTensor(self.cholesky())
   1003             return cholesky.inv_quad_logdet(inv_quad_rhs=inv_quad_rhs, logdet=logdet, reduce_inv_quad=reduce_inv_quad)
   1004 

~\anaconda3\lib\site-packages\gpytorch\lazy\lazy_tensor.py in cholesky(self, upper)
    737             (LazyTensor) Cholesky factor (lower triangular)
    738         """
--&gt; 739         res = self._cholesky()
    740         if upper:
    741             res = res.transpose(-1, -2)

~\anaconda3\lib\site-packages\gpytorch\utils\memoize.py in g(self, *args, **kwargs)
     32         cache_name = name if name is not None else method
     33         if not is_in_cache(self, cache_name):
---&gt; 34             add_to_cache(self, cache_name, method(self, *args, **kwargs))
     35         return get_from_cache(self, cache_name)
     36 

~\anaconda3\lib\site-packages\gpytorch\lazy\lazy_tensor.py in _cholesky(self)
    412 
    413         # contiguous call is necessary here
--&gt; 414         cholesky = psd_safe_cholesky(evaluated_mat).contiguous()
    415         return NonLazyTensor(cholesky)
    416 

~\anaconda3\lib\site-packages\gpytorch\utils\cholesky.py in psd_safe_cholesky(A, upper, out, jitter)
     46             except RuntimeError:
     47                 continue
---&gt; 48         raise e

~\anaconda3\lib\site-packages\gpytorch\utils\cholesky.py in psd_safe_cholesky(A, upper, out, jitter)
     23     """
     24     try:
---&gt; 25         L = torch.cholesky(A, upper=upper, out=out)
     26         return L
     27     except RuntimeError as e:

RuntimeError: cholesky_cpu: U(135,135) is zero, singular U.
</code></pre>
<p>What can I do?<br>
Thanks</p>

KFrank: <p>Hi RR!</p>
<aside class="quote no-group" data-username="RR_1" data-post="1" data-topic="94589" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rr_1/48/11836_2.png" class="avatar"> RR_1:</div>
<blockquote>
<pre><code class="lang-auto">RuntimeError: cholesky_cpu: U(135,135) is zero, singular U.
</code></pre>
</blockquote>
</aside>
<p>It looks like this is a know bug – see the following two github issues:</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/cornellius-gp/gpytorch/issues/943">
  <header class="source">

      <a href="https://github.com/cornellius-gp/gpytorch/issues/943" target="_blank" rel="noopener nofollow ugc">github.com/cornellius-gp/gpytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/cornellius-gp/gpytorch/issues/943" target="_blank" rel="noopener nofollow ugc">[Bug] cholesky singular U</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-11-14" data-time="04:24:31" data-timezone="UTC">04:24AM - 14 Nov 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-11-16" data-time="08:55:38" data-timezone="UTC">08:55AM - 16 Nov 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/jeffwillette" target="_blank" rel="noopener nofollow ugc">
          <img alt="jeffwillette" src="https://avatars.githubusercontent.com/u/9424192?v=4" class="onebox-avatar-inline" width="20" height="20">
          jeffwillette
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          bug
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          variational
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container"># 🐛 Bug



## To reproduce

I followed the directions here for a variation<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">al GP (https://gpytorch.readthedocs.io/en/latest/examples/05_Scalable_GP_Regression_Multidimensional/SVGP_Regression_CUDA.html#Overview) with the HEAD build of gpytorch and it trains for a while and then crashes with...

** Stack trace/error message **
```
  File "main.py", line 93, in &lt;module&gt;
    pred = model(x)███████████████████████████████████████████████████████████████████████▉       | 999/1088 [00:42&lt;00:02, 31.22it/s]
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/models/approximate_gp.py", line 56, in __call__
    return self.variational_strategy(inputs, prior=prior)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/variational/whitened_variational_strategy.py", line 240, in __call__
    return Module.__call__(self, x)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/module.py", line 24, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/variational/whitened_variational_strategy.py", line 141, in forward
    induc_induc_covar = CholLazyTensor(induc_induc_covar.cholesky())                                                                
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/lazy/lazy_tensor.py", line 738, in cholesky
    res = self._cholesky()
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/utils/memoize.py", line 34, in g
    add_to_cache(self, cache_name, method(self, *args, **kwargs))
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/lazy/lazy_tensor.py", line 413, in _cholesky
    cholesky = psd_safe_cholesky(evaluated_mat).contiguous()
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/utils/cholesky.py", line 39, in psd_safe_cholesky
    raise e
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/utils/cholesky.py", line 22, in psd_safe_cholesky
    L = torch.cholesky(A, upper=upper, out=out)
RuntimeError: cholesky_cpu: U(1,1) is zero, singular U.
```

## Expected Behavior

I'm not sure why it is running into a singular matrix there, as I would expect that case to be covered. 

## System information

**Please complete the following information:**
0.3.6 - cd1c8f8
1.2.0
Ubuntu 

## Additional context

Additionally, I get a cuda OOM error every time I try to call `.to(device)` using gpytorch, even with minimal data, which is strange.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/botorch/issues/179">
  <header class="source">

      <a href="https://github.com/pytorch/botorch/issues/179" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/botorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/botorch/issues/179" target="_blank" rel="noopener nofollow ugc">Numerical issue with cholesky decomposition (even with normalization)</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-06-23" data-time="22:06:33" data-timezone="UTC">10:06PM - 23 Jun 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-07-03" data-time="04:41:55" data-timezone="UTC">04:41AM - 03 Jul 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/michaelyli" target="_blank" rel="noopener nofollow ugc">
          <img alt="michaelyli" src="https://avatars.githubusercontent.com/u/29965467?v=4" class="onebox-avatar-inline" width="20" height="20">
          michaelyli
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## Issue description
I am consistently running into numerical issues when runni<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">ng fit_gpytorch_model(). I am normalizing the inputs and standardizing the outputs (as described in issue 160)

## Code example
    fit_gpytorch_model(mll)
    EI = ExpectedImprovement(gp, best_f=0.1)
    
    ## optimize acquisition function
    candidates = joint_optimize(
        acq_function=EI,
        q = 1, 
        bounds = bounds,
        num_restarts=10,
        raw_samples=500,  # used for intialization heuristic
    )
    
    new_x = candidates.detach()
    exact_obj = neg_eggholder(new_x)
    
    train_x_ei = torch.cat([train_x_ei, candidates])
    train_y_ei = torch.cat([train_y_ei, exact_obj])
   
    gp = SingleTaskGP(
        normalize(train_x_ei, bounds=bounds), 
        standardize(train_y_ei)
        )
    mll = ExactMarginalLogLikelihood(gp.likelihood, gp)


Here is the error message:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-23-93bb2f6695a0&gt; in &lt;module&gt;
      2 
      3 
----&gt; 4     fit_gpytorch_model(mll)
      5     EI = ExpectedImprovement(gp, best_f=0.1)
      6 

C:\Anaconda3\envs\py36_new\lib\site-packages\botorch\fit.py in fit_gpytorch_model(mll, optimizer, **kwargs)
     33     """
     34     mll.train()
---&gt; 35     mll, _ = optimizer(mll, track_iterations=False, **kwargs)
     36     mll.eval()
     37     return mll

C:\Anaconda3\envs\py36_new\lib\site-packages\botorch\optim\fit.py in fit_gpytorch_scipy(mll, bounds, method, options, track_iterations)
    186         jac=True,
    187         options=options,
--&gt; 188         callback=cb,
    189     )
    190     iterations = []

C:\Anaconda3\envs\py36_new\lib\site-packages\scipy\optimize\_minimize.py in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
    599     elif meth == 'l-bfgs-b':
    600         return _minimize_lbfgsb(fun, x0, args, jac, bounds,
--&gt; 601                                 callback=callback, **options)
    602     elif meth == 'tnc':
    603         return _minimize_tnc(fun, x0, args, jac, bounds, callback=callback,

C:\Anaconda3\envs\py36_new\lib\site-packages\scipy\optimize\lbfgsb.py in _minimize_lbfgsb(fun, x0, args, jac, bounds, disp, maxcor, ftol, gtol, eps, maxfun, maxiter, iprint, callback, maxls, **unknown_options)
    333             # until the completion of the current minimization iteration.
    334             # Overwrite f and g:
--&gt; 335             f, g = func_and_grad(x)
    336         elif task_str.startswith(b'NEW_X'):
    337             # new iteration

C:\Anaconda3\envs\py36_new\lib\site-packages\scipy\optimize\lbfgsb.py in func_and_grad(x)
    283     else:
    284         def func_and_grad(x):
--&gt; 285             f = fun(x, *args)
    286             g = jac(x, *args)
    287             return f, g

C:\Anaconda3\envs\py36_new\lib\site-packages\scipy\optimize\optimize.py in function_wrapper(*wrapper_args)
    298     def function_wrapper(*wrapper_args):
    299         ncalls[0] += 1
--&gt; 300         return function(*(wrapper_args + args))
    301 
    302     return ncalls, function_wrapper

C:\Anaconda3\envs\py36_new\lib\site-packages\scipy\optimize\optimize.py in __call__(self, x, *args)
     61     def __call__(self, x, *args):
     62         self.x = numpy.asarray(x).copy()
---&gt; 63         fg = self.fun(x, *args)
     64         self.jac = fg[1]
     65         return fg[0]

C:\Anaconda3\envs\py36_new\lib\site-packages\botorch\optim\fit.py in _scipy_objective_and_grad(x, mll, property_dict)
    221     output = mll.model(*train_inputs)
    222     args = [output, train_targets] + _get_extra_mll_args(mll)
--&gt; 223     loss = -mll(*args).sum()
    224     loss.backward()
    225     param_dict = OrderedDict(mll.named_parameters())

C:\Anaconda3\envs\py36_new\lib\site-packages\gpytorch\module.py in __call__(self, *inputs, **kwargs)
     20 
     21     def __call__(self, *inputs, **kwargs):
---&gt; 22         outputs = self.forward(*inputs, **kwargs)
     23         if isinstance(outputs, list):
     24             return [_validate_module_outputs(output) for output in outputs]

C:\Anaconda3\envs\py36_new\lib\site-packages\gpytorch\mlls\exact_marginal_log_likelihood.py in forward(self, output, target, *params)
     26         # Get the log prob of the marginal distribution
     27         output = self.likelihood(output, *params)
---&gt; 28         res = output.log_prob(target)
     29 
     30         # Add terms for SGPR / when inducing points are learned

C:\Anaconda3\envs\py36_new\lib\site-packages\gpytorch\distributions\multivariate_normal.py in log_prob(self, value)
    127 
    128         # Get log determininat and first part of quadratic form
--&gt; 129         inv_quad, logdet = covar.inv_quad_logdet(inv_quad_rhs=diff.unsqueeze(-1), logdet=True)
    130 
    131         res = -0.5 * sum([inv_quad, logdet, diff.size(-1) * math.log(2 * math.pi)])

C:\Anaconda3\envs\py36_new\lib\site-packages\gpytorch\lazy\lazy_tensor.py in inv_quad_logdet(self, inv_quad_rhs, logdet, reduce_inv_quad)
    990             from .chol_lazy_tensor import CholLazyTensor
    991 
--&gt; 992             cholesky = CholLazyTensor(self.cholesky())
    993             return cholesky.inv_quad_logdet(inv_quad_rhs=inv_quad_rhs, logdet=logdet, reduce_inv_quad=reduce_inv_quad)
    994 

C:\Anaconda3\envs\py36_new\lib\site-packages\gpytorch\lazy\lazy_tensor.py in cholesky(self, upper)
    716             (LazyTensor) Cholesky factor (lower triangular)
    717         """
--&gt; 718         res = self._cholesky()
    719         if upper:
    720             res = res.transpose(-1, -2)

C:\Anaconda3\envs\py36_new\lib\site-packages\gpytorch\utils\memoize.py in g(self, *args, **kwargs)
     32         cache_name = name if name is not None else method
     33         if not is_in_cache(self, cache_name):
---&gt; 34             add_to_cache(self, cache_name, method(self, *args, **kwargs))
     35         return get_from_cache(self, cache_name)
     36 

C:\Anaconda3\envs\py36_new\lib\site-packages\gpytorch\lazy\lazy_tensor.py in _cholesky(self)
    401             evaluated_mat.register_hook(_ensure_symmetric_grad)
    402 
--&gt; 403         cholesky = psd_safe_cholesky(evaluated_mat.double()).to(self.dtype)
    404         return NonLazyTensor(cholesky)
    405 

C:\Anaconda3\envs\py36_new\lib\site-packages\gpytorch\utils\cholesky.py in psd_safe_cholesky(A, upper, out, jitter)
     45                 continue
     46 
---&gt; 47         raise e
     48 
     49 

C:\Anaconda3\envs\py36_new\lib\site-packages\gpytorch\utils\cholesky.py in psd_safe_cholesky(A, upper, out, jitter)
     19     """
     20     try:
---&gt; 21         L = torch.cholesky(A, upper=upper, out=out)
     22         # TODO: Remove once fixed in pytorch (#16780)
     23         if A.dim() &gt; 2 and A.is_cuda:

RuntimeError: cholesky_cpu: U(2,2) is zero, singular U.

## System Info
BoTorch 0.1.0
GPyTorch 0.3.2
Torch 1.1.0
Windows 10</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Even though these two issues are marked closed, it’s not clear to me<br>
that the core issue has been addressed.</p>
<p>In any event, it appears that some upstream code is letting a singular<br>
matrix escape to a Cholesky-decomposition routine that expects a<br>
positive-definite matrix.</p>
<aside class="quote no-group">
<blockquote>
<p>What can I do?</p>
</blockquote>
</aside>
<p>These two github issues (as well as others) discuss some possible<br>
work-arounds.  You could read through the issue discussions and see<br>
if any of the suggestions work for your use case.</p>
<p>If you’re not running the latest and greatest, you could try upgrading to<br>
the up-to-date version of whatever package you are using and see if<br>
the github issues were, in fact, legitimately closed.</p>
<p>Good luck.</p>
<p>K. Frank</p>