A sparse_matrix can be permuted now provided it is a leaf