Ensure sparse inverse works.

It looks like it might be faster to convert to dense and invert
normally though :-/