Again, this is a candidate for cythonization.
There has been some work in scipy on fancy indexing of sparse matrices... it might be work checking whether something in there would apply here.