Revert "Fix subtle bug when src & dst agree on sparsity but have different dimensions"

This reverts commit 48a3eaf53e7f430d4490f0ac8d6d8ff3c5701b1e.