Make matrix inverses accept unsimplified fractions

Fixes #37449