ENH: avoid repeated matrix inverse

The `sum_cov` term doesn't change, so it's faster to use a factorized version and solve.
Merge pull request #3769 from perimosocordiae/patch-5

ENH: avoid repeated matrix inverse