Using an LDLT_factor here should be possible and will be much faster and more numerically stable.