Don't compute factorization twice

See #4732 and d46936bcc9a55ea184f3420ef900ba7f9ccc83b9. Ugh.