Remove unncessary divisions for mat inverses

There are lot of division that were happening, especially for the mat3 and mat4 cases.  This should speed things up a bit, by preferring multiplications over divisions.

The bottom of the file already checks that the template type T is always a floating point type, so we won't need to worry about the integral division round off.