BUG: sparse matrices don't allow some operator overloads

This seems to fix #3580. It's not entirely finished yet as I want to pull in some patches from @pv and clean some things up.