Get rid of the generic inv implementation as it really is only
applicable for the dense case, which already has its own implementation.
See #4439 also.