Matrix interpolation

> Can this actually happen, or is this a testing-only assert?

Yes this can happen. All you have to do to hit it is put a matrix3d which violates it into one of the keyframes.