ENH: avoid repeated matrix inverse

The `sum_cov` term doesn't change, so it's faster to use a factorized version and solve.